はてなのサーバ管理ツールの話

はてなのサーバ管理ツールの思想とサーバメトリクス可視化システムの実装

Transcript

1. ͸ͯͳͷαʔό؅ཧπʔϧͷ࿩ Mackerel - Server Management with Tool YAPC::Asia Tokyo 2013
2. Hatena
   JEZ@VVLJ Twitter
   !Z@VVL CPAN
   
   
   :66,* ͸ͯͳࣾΞϧόΠτࢯ !γεςϜϓϥοτϑΥʔϜ෦
3. ࠓ೔ͷ༷ࢠͰ͢
4. ͍͖͞Μ৬࣭͞Ε·ͨ͠ http://moznion.hatenadiary.com/entry/2013/08/14/010104 id:moznionࢯ
5. Perlྺ1೥͘Β͍
6. Perl CPAN WebService::Qiita RRDTool::Rawish Plack::Middleware::GitStatus Net::Signalet vim-plugin unite-perl-module.vim perl-local-lib-path.vim
7. Kyoto.pm
8. YAPCࢀՃ͸͡ΊͯͰ͢
9. ͻʔ y_uuki
10. 40෼͠Ό΂Γ·͢
11. ͦͷ·͑ʹ
12. ӡ༻ܥͷํʁ
13. Muninͱ͔Zabbixܥͷπʔϧ Λ࢖ͬͯΒͬ͠ΌΔํʁ
14. RRDtoolΛ͝ଘ஌ͷํ?
15. ࠓ೔ͷ͓࿩ ͸ͯͳͷαʔό؅ཧπʔϧͷࢥ૝ ύϑΥʔϚϯεϝτϦΫε ՄࢹԽͷ࢓૊Έͷ࣮૷
16. αʔό؅ཧπʔϧ Munin Cacti Zabbix Ganglia collectd etc...
17. ͍Ζ͍Ζ͋Δ
18. αʔό؅ཧपลπʔϧ ؂ࢹܥ ߏ੒؅ཧܥ σϓϩΠܥ Cinnamon
19. ͍Ζ͍Ζ͋Δ
20. ໰୊఺
21. ϗετ৘ใ͕෼ࢄͯ͠͠·͏ ໨తผʹπʔϧ͕ଘࡏ͢Δ Nagiosͷconfig Capistranoͷconfig.rb Muninͷconfig Chef Server
22. ͡Ό͋ϑϧελοΫܥͷ πʔϧ࢖͑͹?
23. ϑϧελοΫπʔϧ Zabbix (ϗετ؅ཧ + ؂ࢹ) Crowbar (Ganglia + Chef + Nagios) ࢖͍͍ͨπʔϧͱ૊Έ߹Θͤʹ͍͘ ֤ػೳ͕ີ݁߹
24. ϗετ৘ใΛ෼ࢄͤͣ͞ҰݩԽ͍ͨ͠ ֤πʔϧͱૄ݁߹ʹ౷߹͍ͨ͠ ࣗ࡞͠Α͏ʂ
25. ϑϧεΫϥον http://anond.hatelabo.jp/20120313004820
26. Mackerel ʲՄࢉ໊ࢺʳɹαό ʲෆՄࢉ໊ࢺʳαό(ͷ਎) Plack / Starlet / Teng / Router::Simple / Class::Accessor::Lite::Lazy / TheSchwartz / Text::Xslate / Config::ENV / Scope::Container / SQL::Maker ... Perl 5.14 / Carton 1.0
27. ·ͣ͸ࢥ૝͔Β
28. ϗετ৘ใͷҰݩ؅ཧ ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍ ֤छπʔϧͱૄ݁߹ʹ౷߹ REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
29. ϗετ৘ใͷҰݩ؅ཧ ϗετ؅ཧ ෺ཧϗετ or Ծ૝ϗετ or AWS ෺ཧϗετͱԾ૝ϗετͷରԠؔ܎ ͲͷαʔϏεͷͲ͏͍͏໾ׂ(DBαʔό,ϓϩΩγαʔόͳͲʣ ͲͷσʔληϯλͷͲͷϥοΫ ͲͷωοτϫʔΫʢαϒωοτʣ ঢ়ଶ (working,standby,maintenace,poweroff,destroyed)
30. ServiceҰཡ
31. ServiceҎԼͷRoleҰཡ RoleҰཡ Role ϗετ৘ใ
32. ৄࡉϗετ৘ใ load average CPU time iops memory usaage ... other graphs
33. पลπʔϧͱͷ࿈ܞ
34. Nagios࿈ܞ
35. ઃఆͷࣗಈੜ੒ MackerelͷAPI͔Βϗετ৘ใΛऔಘͯ͠ϗετఆٛ ϑΝΠϧΛࣗಈੜ੒ NagiosͷϗετstatusΛMackerelͷϏϡʔʹ΋දࣔ NagiosͷWeb UIʹͱ͹ͳ͍͍ͯ͘Α͏ʹ ৄ͘͠͸ίνϥ ϞχΧδ#3 http://www.slideshare.net/shoichimasuhara/3-ss-17097401 Nagios࿈ܞ
36. Chef࿈ܞ (Just Idea)
37. Chef Serverͱͷಉظ MackerelͷDBΛϚελͱ͢Δ Mackerelʹߋ৽ɾ࡟আ͕૸Ε͹Chef ServerͷAPIΛୟ ͍ͯPOST Chef Serverͷഇࢭ MackerelͷRole͝ͱʹϨγϐΛ༻ҙ Ϩγϐͷద༻ઌ͸MackerelͷAPI͔Β౎౓औಘ Knife SoloͰϨγϐద༻
38. Capistrano࿈ܞ
39. Capistrano࿈ܞ Capfileʹ௚ʹσϓϩΠઌϦετΛॻ͖ͨ͘ͳ͍ σϓϩΠ͢ΔͨͼʹMackerelͷAPIΛୟ͍ͯσϓϩΠઌ ϗετҰཡΛऔಘ http://mackerelhost/service/Hatena-Blog/roles/ backend?format=json backend = ΞϓϦέʔγϣϯαʔό ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷΑ͏ʹRoleͰࢦఆ
40. IRC࿈ܞ
41. IRC࿈ܞ ϗετ৘ใͷ࡞੒ɾߋ৽ɾ࡟আΛIkachanܦ༝Ͱ IRCʹ௨஌ ୭͕ͲͷΑ͏ͳૢ࡞Λ͔͕ͨ͠ϦΞϧλΠϜʹΘ͔Δ DBૢ࡞ʹରͯ͠௨஌ͷHookΛ࢓ࠐΉ (wtatsuru) update host 'intern2013': {"status"=>"maintenance"} εςʔλεΛ”ϝϯςφεத”ʹҠߦ
42. DNS࿈ܞ
43. RoleΛߏ੒͢Δϗετ reverse proxy LVS Role Role ৑௕Խɾෛՙ෼ࢄ͞Ε͍ͯΔ͜ͱʹண໨
44. ৑௕Խɾෛՙ෼ࢄ Role Ծ૝IP(VIP) LVS Role DNS ϥ΢ϯυϩϏϯ DNS DNSϥ΢ϯυϩϏϯ༻FQDNʹର ͯ͠ෳ਺ͷIP͔Βϥ΢ϯυϩϏ ϯͰҰͭʹIPΛฦ͢ LVSϗετʹԾ૝IPΛׂΓ౰ͯ Ծ૝IP͑͞Θ͔͍ͬͯΕ͹ɹɹ ͋ͱ͸উखʹ෼ࢄ
45. ϥ΢ϯυϩϏϯ༻FQDNͱ VIP͸͋·Γม͑ͨ͘ͳ͍ Service໊ͱRole໊͸΄ͱΜͲมΘΒ ͳ͍͜ͱʹண໨
46. DNSϥ΢ϯυϩϏϯ DNSϥ΢ϯυϩϏϯ༻ͷFQDNΛService-Roleϕʔε ͷ໊લͰׂΓ౰ͯ nslookup hosts.db-slave.bookmark Name: hosts.db-slave.bookmark Address: 10.0.0.1 Name: hosts.db-slave.bookmark Address: 10.0.0.2 Name: hosts.db-slave.bookmark Address: 10.0.0.3 role.service
47. VIP LVSͷVIPʹରͯ͠FQDNΛService-Roleϕʔε ͷ໊લͰׂΓ౰ͯ nslookup db-slave.bookmark Name: db-slave.bookmark Address: 192.168.1.1 role.service
48. पลπʔϧͱͷ࿈ܞͷ࿩ऴΘΓ
49. αʔό؅ཧʹ޲͍ͨϏϡʔ
50. ৘ใີ౓͕ߴ͘໢ཏతͳϏϡʔ ڝഅ৽ฉΈ͍ͨͳΠϝʔδ ۭ͖༰ྔͱ͔͕ͬ͟ͱΈΕΔ
51. ϗετ໊/IP ͳͲ loadavg άϥϑ Datacenter/ Rack Service/ Role Resource Memo
52. ͍ͭ΋ΈΔάϥϑ΍ൺֱ͍ͨ͠άϥϑΛબΜͰ̍ϖʔδ ʹ·ͱΊΔ HTMLʹՃ͑ͯɺάϥϑه๏Λ࢖ͬͯΧελϚΠζ <a href="/services/SERVICE/roles/ROLE/hosts"> <img src="/services/SERVICE/roles/ROLE/graph"> </a> [:graph:service=SERVICE:role=ROLE] [:graph:service=SERVICE:role=ROLE] <iframe src="/services/SERVICE/roles/ROLE/hosts? format=embed"></iframe> ΧελϚΠζͰ͖ΔμογϡϘʔυ
54. Roleϗετͷάϥϑൺֱ ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷload averageൺֱ ҟৗͳϗετ͕͍ͳ͍͔Ͳ͏͔ݕग़
55. લ൒͓ΘΓ
56. αʔόϝτϦΫεՄࢹԽͷ ࢓૊Έ ϝτϦΫε
    MPBE
    BWFSBHFɼ$16
    UJNFɼNFNPSZ
    VTBHF
    FUD
57. αʔόϝτϦΫεՄࢹԽͬͯʁ
59. MackerelͰͲ͏΍࣮ͬͯ૷ɹ ͯ͠Δ͔ͷ࿩
60. αʔόϝτϦΫεՄࢹԽʹඞਢͷཁૉ
61. αʔόϝτϦΫεՄࢹԽ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ ετϨʔδʹ֨ೲ ετϨʔδ͔Β஋ΛऔΓग़͢ άϥϑԽ
62. ͲͷΑ͏ͳΞʔΩςΫνϟ͕Α͍͔
63. Push or Pull
64. Pushܕ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ Host͔ΒProcess΁ ϝτϦΫεΛPush
65. Pushܕͷ໰୊఺ ֤ϗετʹcronͰճ͢ϝτϦΫεऔಘ&ૹ৴εΫ ϦϓτΛ഑෍͠ͳ͚Ε͹ͳΒͳ͍ εΫϦϓτͷܧଓతͳมߋ͕໘౗
66. Pullܕ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ Process͔ΒHost΁ ϦΫΤετ͢Δ
67. PullܕͰ΋݁ہࣗલͷ ʢܧଓతͳมߋ͕ඞཁͳʣσʔϞϯΛ ཱͯͳ͚Ε͹ͳΒͳ͍ͷͰ͸ʁ
68. ϝτϦΫεऔಘͷͨΊͷ ωοτϫʔΫΠϯλϑΣʔεΛ΋ͭ طଘͷσʔϞϯΛ࢖͏
69. SNMP OS͕ఏڙ͢ΔجຊతͳϝτϦΫεΛऔಘ ͦͷଞϧʔλ΍SquidͳͲ Net::SNMPΛ࢖͏ my ($session, $error) = Net::SNMP->session( -hostname => $hostname, -community => $community, -version => 2, -timeout => 10, -translate => 0x0, ); $session // croak "SNMP error: $error"; my $response = $session->get_request( -varbindlist => $mibs, # MIBΛෳ਺ࢦఆ ) || croak "SNMP error: $session->error";
70. Nginx HttpStubStatusModule HTTPͰNginxݻ༗ͷϝτϦΫεΛͱΕΔ Active connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 $ curl http://nginxhost/nginx_status:8080
71. Plack Plack::Middleware::ServerStatus::Lite JSON format͕͋ΔͷͰָ % curl http://server:port/server-status?json {"Uptime":"1332476669","BusyWorkers":"2", "stats":[ {"protocol":null,"remote_addr":null,"pid":"78639", "status":".","method":null,"uri":null,"host":null,"ss":null}, {"protocol":"HTTP/ 1.1","remote_addr":"127.0.0.1","pid":"78640", "status":"A","method":"GET","uri":"/","host":"localhost: 10226","ss":0}, ... ],"IdleWorkers":"3"}
72. Memcached TelnetΠϯλϑΣʔε stats STAT pid 14868 STAT uptime 175931 STAT time 1220540125 STAT version 1.2.2 ... STAT curr_connections 92 STAT total_connections 1740 STAT connection_structures 165 STAT cmd_get 7411 STAT cmd_set 28445156 STAT get_hits 5183 STAT get_misses 2228 STAT evictions 0 STAT bytes_read 2112768087 STAT bytes_written 1000038245 STAT limit_maxbytes 52428800 STAT threads 1 END my $sock = IO::Socket::INET->new( PeerAddr => "$hostname:$port", Proto => 'tcp', Timeout => 10, ) or croak "Couldn't connect to $hostname:$port"; $sock->print("stats\r\n"); my $value_by_stat = {}; while (my $line = $sock->getline) { last if $line =~ /^END/; $line =~ s/\n$|\r\n$//; #chomp if ($line =~ /^STAT\s+(\S*)\s+(.*)/) { $value_by_stat->{$1} = $2; } }
73. Pullܕ (࠶ܝ) Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ ετϨʔδʹ֨ೲ ετϨʔδ͔Β஋ΛऔΓग़͢
74. Process
75. Process ϗετ͸ͨ͘͞Μ͋ΔͷͰฒྻ(ඇಉظ)Ͱɹɹɹ ϝτϦΫεऔಘ͍ͨ͠ ͔͠͠ɺϗετ਺෼ͷϓϩηεΛཱͯΔͷ͸ ͪΐͬͱ... PerlͰεϨου...
76. WorkerManager Job Queue(The Schwartz)ʹରԠͨ͠ϓϩηε؅ཧ σʔϞϯ Preforkܕ: ϓϩηε਺Λ੍ݶͰ͖Δ ͔ΘΓʹParallel::Prefork΍Starlet/Starman Λ࢖ͬͯ΋Α͍
77. Jobͷॲཧ಺༰
78. Agent ֤JobΛAgentͱݺΜͰ͍Δ ϦϞʔτϗετ͔ΒϝτϦΫεΛͱͬͯ͘Δϓϩηε ରԠϛυϧ΢ΣΞɼϓϩτίϧ Apache, Nginx, MySQL, Munin, Latency, Plack, Perlbal, Redis, SNMP, Solr, TheSchwartz ϓϥΨϒϧ Hatena::Mackerel::Worker::Agent::XXX SNMP΍HTTP, TelnetͰϝτϦΫεऔಘ
79. ϓϥάΠϯ͸PerlͰͳ͍ͱॻ͚ͳ͍ ݴޠඇґଘͰϓϥάΠϯΛॻ͖͍ͨ
80. ͦΕMunin͡ΌͶ MuninͷϓϥάΠϯ͸ ඪ४ೖग़ྗͰύϥϝʔλͷड͚౉͠Λ͢Δ
81. munin-nodeͱͷ࿈ܞ munin-nodeΛϦϞʔτϗετʹཱ͓ͯͯ͘ mackerel agent͕TelnetͰmunin pluginͷ݁Ռऔಘ munin-node munin plugin mackerel agent telnet ඪ४ೖग़ྗ
82. munin-nodeͱซ༻ ϛυϧ΢ΣΞ͕ϝτϦΫεऔಘͷͨΊͷωοτϫʔ ΫΠϯλϑΣʔε͕͋ΔͳΒ͹ mackerel agent͔ΒωοτϫʔΫɾΠϯλ ϑΣʔεΛ࢖͏ netstat΍smartctlͳͲϦϞʔτϗετͷίϚϯ υΛୟ͖͍ͨͳΒ͹ munin-nodeΛ࢖͏
83. ϝτϦΫεͷߋ৽ (ετϨʔδͷ࿩)
84. Pullܕ (࠶ܝ) Host Host Host Host Worker Manager Storage
85. Storage RDBMS, KVS, other NOSQL...
86. RRDtool
87. RRD (Round Robin Database) ࣌ܥྻσʔλΛ֨ೲ͢ΔͨΊͷσʔλϑΥʔϚοτ อଘظؒΛݻఆͯ͠ݹ͍஋͸ࣺ͍ͯͯ͘ RRDtool RRDͷ࣮૷ʢσϑΝΫτελϯμʔυʣ RRDͷ࡞੒ɾߋ৽ɾࢀরػೳͱάϥϑԽػೳΛ΋ͭ I/O͕΍ͨΒ଎͍(Β͍͠) େ఍ͷαʔό؅ཧπʔϧͰ࢖ΘΕ͍ͯΔ ͳ͔ͥάϥϑը૾ੜ੒ػೳ͕͋Δ
88. RRDtoolϝϦοτɾσϝϦοτ ϝϦοτ ࣌ܥྻσʔλΛ͍͍͔Μ͡ʹॻ͖ࠐΜͰ͘ΕΔ ߋ৽ִ͔ؒΒ࣮ࡍͷॻ͖ࠐΈ࣌ࠁ͕গʑͣΕͯ΋मਖ਼ͯ͘͠ ΕΔ (10:01 -> 10:00) ஋ͷਫ਼౓ͷௐઅ͕Մೳ ϥ΢ϯυϩϏϯͳͷͰϑΝΠϧαΠζ͕Ұఆ I/O͕ߴ଎ σϝϦοτ rrdtoolʹ༩͑Δύϥϝʔλͷઃఆ͕൥ࡶ
89. ొ৔ਓ෺͕ͰͦΖͬͨͷͰ γεςϜશମΛ੔ཧ
90. Job Queue DB RRDtool Host Host Host Host enqueue.pl cron δϣϒ౤ೖ ϗετ৘ใऔಘ ϝτϦΫεऔಘ
91. RRDtool App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ
92. RRDtool App Server Browser HTTP graph response graph image άϥϑϨεϙϯε
93. Job Queue DB RRD Host Host Host Host enqueue.pl cron RRDtool App Server Browser γεςϜશମ
94. Web͔Βάϥϑੜ੒ͤ͞Δ ࢓૊Έ
95. RRD App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ ʢ࠶ܝʣ
96. RRD App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ ʢ࠶ܝʣ άϥϑੜ੒URLΛͲ͏දݱ ͢Δ͔
97. ཁ͸rrdtool graphίϚϯυͷҾ਺ ΛURLͰͲ͏දݱ͢Δ͔
98. rrdtool graph $ rrdtool graph --end now --start end-120000s --width 400 \\ DEF:ds0a=/home/rrdtool/data/router1.rrd:ds0:AVERAGE \\ DEF:ds0b=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=1800 \\ DEF:ds0c=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=7200 \\ LINE1:ds0a#0000FF:"default resolution\l" \\ LINE1:ds0b#00CCFF:"resolution 1800 seconds per interval\l" \\ LINE1:ds0c#FF00FF:"resolution 7200 seconds per interval\l" άϥϑඳըظؒͷࢦఆ άϥϑ෯ࢦఆ σʔλϦιʔεʢRRDϑΝΠϧ໊ʣࢦఆ άϥϑઢͷଠ͞΍৭ɺຌྫΛࢦఆ ...
99. γϯϓϧURL ͱ RRDtool URL ͷ2छྨ༻ҙ
100. γϯϓϧURL /graph? define=tesuhost01::_default::loadavg5::STACK,testhost0 2::mysql::second_behind_master define=<host>::<tag>::<metrics name>, define=<host>::<tag>::<metrics name>, ... σϑΥϧτ஋ΛܾΊͯɺͳΔ΂͘γϯϓϧʹ ઢͷଠ͞ͱ͔͸σϑΥϧτ஋Λ಺෦ͰܾΊ͓ͯ͘
101. RRDtool URL RRDtoolίϚϯυΛͦͷ··URDͰදݱ hit rateͷܭࢉͱ͔Λͤ͞Δͱ͖ʹ࢖͏ /graph/general?s=[ (def:cmd_get:::=path:memcachedhost,memcached,cmd_get:value:AVERAGE), (def:get_hits:::=path:memcachedhost,memcached,get_hits:value:AVERAGE), (cdef:hit_rate:::=get_hits,cmd_get,/), (line1:hit_rate:::@0000ff:memcachedhost-memcached-hit_rate) !end=now-0d,height=100,start=now-30d,width=400 ] memcachedͷhit rateάϥϑURL
102. my $fmt = <<'EOS'; graphformula = "[graph:" ( graphs [ ":" option ] ) "]" graphs = graph | "(" graph ")" 0*( ",(" graph ")" ) graph = "path:" elements | instruction ":" [ name ] ":::" option ) elements = object : [ tag : [ label ] ] object = name tag = name label = name instruction = "def" | "cdef" | "vdef" | "line" digit | "area" | "hrule" | "vrule" | "print" | "gprint" | "commend" | "tick" | "shift" | "textalign" option = 1*( char | "," | "=" | "#" | "@" | ":" | " " | "\" | op) name = 1*char char = alphanum | mark op = "+" | "/" | "%" | "*" mark = "-" | "_" | "." | "!" | "~" | "*" | "'" alphanum = alpha | digit alpha = lowalpha | upalpha lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" upalpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z" digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" EOS BNFͰਅ໘໨ʹ࢓༷Λఆٛ
103. ύϑΥʔϚϯεվળ
104. RRDͷσΟεΫI/Oෛՙ (ϗετ਺) x (ϝτϦΫεछผ਺) io per 5෼ ϗετ਺ = ਺ઍ ϝτϦΫεछผ਺ = 20Ҏ্ 10k io per 5෼ ࣮ࡍ͸5෼ͷ͏ͪͷ͋Δ࣌ؒʹI/O͕ूத
105. 2छྨͷෛՙ࡟ݮํ๏
106. rrdcached rrdtoolͷߋ৽΍ू໿ॲཧΛड͚෇͚ΔσʔϞϯ rrdͷߋ৽͸ಉ࣌͡ࠁʹࡴ౸͕ͪ͠ ॻ͖ࠐΈΛϥϯμϜͳ࣌ؒ෼஗Ԇͯ͘͠ΕΔ rrdcachedͳͩ͠ͱ, ฏۉ 700 iops όʔετ஋ 1200 iops SSDͩͱͳΜͱ͔ͳΔ HDD(AWS)ͩͱݫ͍͠ ϦϦʔε൛(1.4.8)͸updateʹ͔͠ରԠ͍ͯ͠ͳ͍ ͕trunkͳΒ͹ͦͷଞͷ֤छαϒίϚϯυʹରԠ
107. ෼ࢄRRD RRDͷσΟεΫI/OΛϦϞʔτϗετ୯ҐͰ෼ࢄ ͋Δϗετ͕ͲͷRRDϗετʹׂΓ౰ͯΒΕ͍ͯΔ ͔ͷ৘ใΛDBʹ͓࣋ͬͯ͘ Job Queue DB RRD enqueue.pl Job Queue RRD Job Queue RRD
108. Future Work
109. mackereld
110. mackereld ϝτϦΫεऔಘͷͨΊͷσʔϞϯ munin-node૬౰ͷػೳ REST APIͰϝτϦΫεऔಘ ϦϞʔτϗετͷϩάΛWebͰϦΞϧλΠϜɹɹ ʹΈΕΔ ֤छϩάϑΝΠϧΛ؂ࢹ͠ɺมߋΛݕ஌ͨ͠Β Server-Sent EventsͰͱ͹͢ ܧଓతͳมߋ͕΍͸ΓΊΜͲ͏
111. RRDtoolʹΑΔ༧ଌάϥϑ
112. RRDtoolʹΑΔ༧ଌάϥϑ σΟεΫ͕ຬഋʹͳΔͷ͸͍ͭʁ 2014೥4݄
113. $ rrdtool graph ./dskUsed.png \ --start="now-1y" \ --end="now+2y" \ --imgformat=PNG \ --title="Disk Usage" \ --height=200 \ --width=500 \ --lower-limit=0 \ DEF:usage=testhost__dskUsed.rrd:value:MAX \ DEF:total=testhost__dskTotal.rrd:value:MAX \ CDEF:c_warn=total,0.85,* \ CDEF:c_crit=total,0.95,* \ VDEF:v_total=total,MINIMUM \ VDEF:v_warn=c_warn,MINIMUM \ VDEF:v_crit=c_crit,MINIMUM \ VDEF:v_usage_slope=usage,LSLSLOPE \ VDEF:v_usage_intercept=usage,LSLINT \ CDEF:c_usage_predict=usage,POP,v_usage_slope,COUNT,*,v_usage_intercept,+ \ CDEF:c_rwarn=c_usage_predict,v_warn,v_total,LIMIT \ VDEF:v_rwarn=c_rwarn,FIRST \ CDEF:c_rcrit=c_usage_predict,v_crit,v_total,LIMIT \ VDEF:v_rcrit=c_rcrit,FIRST \ HRULE:v_warn#FF8800:"warning":dashes=5 \ VRULE:v_rwarn#FF8800::dashes=5 \ HRULE:v_crit#FF4400:"critical" \ VRULE:v_rcrit#FF4400 \ HRULE:v_total#FF0000:"total" \ AREA:usage#00FF00:"Disk Usage" \ LINE1:c_usage_predict#0000FF:"Predict" \ GPRINT:v_rwarn:"Reach warning (85%)\: %c":strftime \ GPRINT:v_rcrit:"Reach critical (95%)\: %c":strftime shoichimasuhara++
114. Rich Visualization
115. Rich Visualization ত࿨ͬΆ͍(ը૾) ฏ੒ੜ·Ε͸ͬͪ͜(SVG) RRDtool D3.js
116. D3.js NVD3.js (Re-usable charts for d3.js) Rickshaw (time-series graph) rrdtool xport --json ͰJSONܗࣜͰ஋Λऔಘɹɹ Ͱ͖ΔͨΊɺετϨʔδ͸RRDͷ·· D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS.
117. RRDtool͔Βͷ୤٫
118. ୅ΘΓʹRedis΍MongoDBͳͲͷετϨʔδΛ࢖͏ FnordMetrics (http://fnordmetric.io/) GrowthForecastʹࣅͯΔ ετϨʔδɿRedis άϥϑɿRickshaw my $furl = Furl->new(timeout => 10); my $res = $furl->post( "http://fnordmetricshost:4242/events", ['Content-Type' => 'application/json'], encode_json({ "_type" => '_incr', "value" => scalar @$statuses, "gauge" => "tweets_per_minute" }), );
119. αʔόϝτϦΫεՄࢹԽͷ࿩ ͓ΘΓ
120. ·ͱΊ
121. ϗετ৘ใͷҰݩ؅ཧ ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍ ֤छπʔϧͱૄ݁߹ʹ౷߹ REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
122. ϝτϦΫεՄࢹԽͷ࣮૷ PullܕΞʔΩςΫνϟ WorkerManagerͰϓϩηε؅ཧ Storageʹ͸RRDtoolΛ࢖͏ RRDtool΂ΜΓ RRDtool͍ͨ΁Μ RRDtool͸΍͍ RRDtoolΉ͔͍ͣ͠
123. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ Thanks to id:stanaka, id:halfrack, id:wtatsuru, id:shoichimasuhara, id:hagihala