はてなのサーバ管理ツールの話

 はてなのサーバ管理ツールの話

はてなのサーバ管理ツールの思想とサーバメトリクス可視化システムの実装

A658ec7f1badf73819dfa501165016c1?s=128

Yuuki Tsubouchi (yuuk1)

September 20, 2013
Tweet

Transcript

  1. ͸ͯͳͷαʔό؅ཧπʔϧͷ࿩ Mackerel - Server Management with Tool YAPC::Asia Tokyo 2013

  2. HatenaJEZ@VVLJ Twitter!Z@VVL CPAN:66,* ͸ͯͳࣾΞϧόΠτࢯ !γεςϜϓϥοτϑΥʔϜ෦

  3. ࠓ೔ͷ༷ࢠͰ͢

  4. ͍͖͞Μ৬࣭͞Ε·ͨ͠ http://moznion.hatenadiary.com/entry/2013/08/14/010104 id:moznionࢯ

  5. Perlྺ1೥͘Β͍

  6. Perl CPAN WebService::Qiita RRDTool::Rawish Plack::Middleware::GitStatus Net::Signalet vim-plugin unite-perl-module.vim perl-local-lib-path.vim

  7. Kyoto.pm

  8. YAPCࢀՃ͸͡ΊͯͰ͢

  9. ͻʔ y_uuki

  10. 40෼͠Ό΂Γ·͢

  11. ͦͷ·͑ʹ

  12. ӡ༻ܥͷํʁ

  13. Muninͱ͔Zabbixܥͷπʔϧ Λ࢖ͬͯΒͬ͠ΌΔํʁ

  14. RRDtoolΛ͝ଘ஌ͷํ?

  15. ࠓ೔ͷ͓࿩ ͸ͯͳͷαʔό؅ཧπʔϧͷࢥ૝ ύϑΥʔϚϯεϝτϦΫε ՄࢹԽͷ࢓૊Έͷ࣮૷

  16. αʔό؅ཧπʔϧ Munin Cacti Zabbix Ganglia collectd etc...

  17. ͍Ζ͍Ζ͋Δ

  18. αʔό؅ཧपลπʔϧ ؂ࢹܥ ߏ੒؅ཧܥ σϓϩΠܥ Cinnamon

  19. ͍Ζ͍Ζ͋Δ

  20. ໰୊఺

  21. ϗετ৘ใ͕෼ࢄͯ͠͠·͏ ໨తผʹπʔϧ͕ଘࡏ͢Δ Nagiosͷconfig Capistranoͷconfig.rb Muninͷconfig Chef Server

  22. ͡Ό͋ϑϧελοΫܥͷ πʔϧ࢖͑͹?

  23. ϑϧελοΫπʔϧ Zabbix (ϗετ؅ཧ + ؂ࢹ) Crowbar (Ganglia + Chef +

    Nagios) ࢖͍͍ͨπʔϧͱ૊Έ߹Θͤʹ͍͘ ֤ػೳ͕ີ݁߹
  24. ϗετ৘ใΛ෼ࢄͤͣ͞ҰݩԽ͍ͨ͠ ֤πʔϧͱૄ݁߹ʹ౷߹͍ͨ͠ ࣗ࡞͠Α͏ʂ

  25. ϑϧεΫϥον http://anond.hatelabo.jp/20120313004820

  26. Mackerel ʲՄࢉ໊ࢺʳɹαό ʲෆՄࢉ໊ࢺʳαό(ͷ਎) Plack / Starlet / Teng / Router::Simple

    / Class::Accessor::Lite::Lazy / TheSchwartz / Text::Xslate / Config::ENV / Scope::Container / SQL::Maker ... Perl 5.14 / Carton 1.0
  27. ·ͣ͸ࢥ૝͔Β

  28. ϗετ৘ใͷҰݩ؅ཧ ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍ ֤छπʔϧͱૄ݁߹ʹ౷߹ REST APIʹΑΔपลπʔϧͱͷ࿈ܞ

  29. ϗετ৘ใͷҰݩ؅ཧ ϗετ؅ཧ ෺ཧϗετ or Ծ૝ϗετ or AWS ෺ཧϗετͱԾ૝ϗετͷରԠؔ܎ ͲͷαʔϏεͷͲ͏͍͏໾ׂ(DBαʔό,ϓϩΩγαʔόͳͲʣ ͲͷσʔληϯλͷͲͷϥοΫ

    ͲͷωοτϫʔΫʢαϒωοτʣ ঢ়ଶ (working,standby,maintenace,poweroff,destroyed)
  30. ServiceҰཡ

  31. ServiceҎԼͷRoleҰཡ RoleҰཡ Role ϗετ৘ใ

  32. ৄࡉϗετ৘ใ load average CPU time iops memory usaage ... other

    graphs
  33. पลπʔϧͱͷ࿈ܞ

  34. Nagios࿈ܞ

  35. ઃఆͷࣗಈੜ੒ MackerelͷAPI͔Βϗετ৘ใΛऔಘͯ͠ϗετఆٛ ϑΝΠϧΛࣗಈੜ੒ NagiosͷϗετstatusΛMackerelͷϏϡʔʹ΋දࣔ NagiosͷWeb UIʹͱ͹ͳ͍͍ͯ͘Α͏ʹ ৄ͘͠͸ίνϥ ϞχΧδ#3 http://www.slideshare.net/shoichimasuhara/3-ss-17097401 Nagios࿈ܞ

  36. Chef࿈ܞ (Just Idea)

  37. Chef Serverͱͷಉظ MackerelͷDBΛϚελͱ͢Δ Mackerelʹߋ৽ɾ࡟আ͕૸Ε͹Chef ServerͷAPIΛୟ ͍ͯPOST Chef Serverͷഇࢭ MackerelͷRole͝ͱʹϨγϐΛ༻ҙ Ϩγϐͷద༻ઌ͸MackerelͷAPI͔Β౎౓औಘ

    Knife SoloͰϨγϐద༻
  38. Capistrano࿈ܞ

  39. Capistrano࿈ܞ Capfileʹ௚ʹσϓϩΠઌϦετΛॻ͖ͨ͘ͳ͍ σϓϩΠ͢ΔͨͼʹMackerelͷAPIΛୟ͍ͯσϓϩΠઌ ϗετҰཡΛऔಘ http://mackerelhost/service/Hatena-Blog/roles/ backend?format=json backend = ΞϓϦέʔγϣϯαʔό ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷΑ͏ʹRoleͰࢦఆ

  40. IRC࿈ܞ

  41. IRC࿈ܞ ϗετ৘ใͷ࡞੒ɾߋ৽ɾ࡟আΛIkachanܦ༝Ͱ IRCʹ௨஌ ୭͕ͲͷΑ͏ͳૢ࡞Λ͔͕ͨ͠ϦΞϧλΠϜʹΘ͔Δ DBૢ࡞ʹରͯ͠௨஌ͷHookΛ࢓ࠐΉ (wtatsuru) update host 'intern2013': {"status"=>"maintenance"}

    εςʔλεΛ”ϝϯςφεத”ʹҠߦ
  42. DNS࿈ܞ

  43. RoleΛߏ੒͢Δϗετ reverse proxy LVS Role Role ৑௕Խɾෛՙ෼ࢄ͞Ε͍ͯΔ͜ͱʹண໨

  44. ৑௕Խɾෛՙ෼ࢄ Role Ծ૝IP(VIP) LVS Role DNS ϥ΢ϯυϩϏϯ DNS DNSϥ΢ϯυϩϏϯ༻FQDNʹର ͯ͠ෳ਺ͷIP͔Βϥ΢ϯυϩϏ

    ϯͰҰͭʹIPΛฦ͢ LVSϗετʹԾ૝IPΛׂΓ౰ͯ Ծ૝IP͑͞Θ͔͍ͬͯΕ͹ɹɹ ͋ͱ͸উखʹ෼ࢄ
  45. ϥ΢ϯυϩϏϯ༻FQDNͱ VIP͸͋·Γม͑ͨ͘ͳ͍ Service໊ͱRole໊͸΄ͱΜͲมΘΒ ͳ͍͜ͱʹண໨

  46. DNSϥ΢ϯυϩϏϯ DNSϥ΢ϯυϩϏϯ༻ͷFQDNΛService-Roleϕʔε ͷ໊લͰׂΓ౰ͯ nslookup hosts.db-slave.bookmark Name: hosts.db-slave.bookmark Address: 10.0.0.1 Name:

    hosts.db-slave.bookmark Address: 10.0.0.2 Name: hosts.db-slave.bookmark Address: 10.0.0.3 role.service
  47. VIP LVSͷVIPʹରͯ͠FQDNΛService-Roleϕʔε ͷ໊લͰׂΓ౰ͯ nslookup db-slave.bookmark Name: db-slave.bookmark Address: 192.168.1.1 role.service

  48. पลπʔϧͱͷ࿈ܞͷ࿩ऴΘΓ

  49. αʔό؅ཧʹ޲͍ͨϏϡʔ

  50. ৘ใີ౓͕ߴ͘໢ཏతͳϏϡʔ ڝഅ৽ฉΈ͍ͨͳΠϝʔδ ۭ͖༰ྔͱ͔͕ͬ͟ͱΈΕΔ

  51. ϗετ໊/IP ͳͲ loadavg άϥϑ Datacenter/ Rack Service/ Role Resource Memo

  52. ͍ͭ΋ΈΔάϥϑ΍ൺֱ͍ͨ͠άϥϑΛબΜͰ̍ϖʔδ ʹ·ͱΊΔ HTMLʹՃ͑ͯɺάϥϑه๏Λ࢖ͬͯΧελϚΠζ <a href="/services/SERVICE/roles/ROLE/hosts"> <img src="/services/SERVICE/roles/ROLE/graph"> </a> [:graph:service=SERVICE:role=ROLE] [:graph:service=SERVICE:role=ROLE]

    <iframe src="/services/SERVICE/roles/ROLE/hosts? format=embed"></iframe> ΧελϚΠζͰ͖ΔμογϡϘʔυ
  53. None
  54. Roleϗετͷάϥϑൺֱ ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷload averageൺֱ ҟৗͳϗετ͕͍ͳ͍͔Ͳ͏͔ݕग़

  55. લ൒͓ΘΓ

  56. αʔόϝτϦΫεՄࢹԽͷ ࢓૊Έ ϝτϦΫεMPBEBWFSBHFɼ$16UJNFɼNFNPSZVTBHF FUD

  57. αʔόϝτϦΫεՄࢹԽͬͯʁ

  58. None
  59. MackerelͰͲ͏΍࣮ͬͯ૷ɹ ͯ͠Δ͔ͷ࿩

  60. αʔόϝτϦΫεՄࢹԽʹඞਢͷཁૉ

  61. αʔόϝτϦΫεՄࢹԽ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ ετϨʔδʹ֨ೲ ετϨʔδ͔Β஋ΛऔΓग़͢

    άϥϑԽ
  62. ͲͷΑ͏ͳΞʔΩςΫνϟ͕Α͍͔

  63. Push or Pull

  64. Pushܕ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ Host͔ΒProcess΁ ϝτϦΫεΛPush

  65. Pushܕͷ໰୊఺ ֤ϗετʹcronͰճ͢ϝτϦΫεऔಘ&ૹ৴εΫ ϦϓτΛ഑෍͠ͳ͚Ε͹ͳΒͳ͍ εΫϦϓτͷܧଓతͳมߋ͕໘౗

  66. Pullܕ Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ Process͔ΒHost΁ ϦΫΤετ͢Δ

  67. PullܕͰ΋݁ہࣗલͷ ʢܧଓతͳมߋ͕ඞཁͳʣσʔϞϯΛ ཱͯͳ͚Ε͹ͳΒͳ͍ͷͰ͸ʁ

  68. ϝτϦΫεऔಘͷͨΊͷ ωοτϫʔΫΠϯλϑΣʔεΛ΋ͭ طଘͷσʔϞϯΛ࢖͏

  69. SNMP OS͕ఏڙ͢ΔجຊతͳϝτϦΫεΛऔಘ ͦͷଞϧʔλ΍SquidͳͲ Net::SNMPΛ࢖͏ my ($session, $error) = Net::SNMP->session( -hostname

    => $hostname, -community => $community, -version => 2, -timeout => 10, -translate => 0x0, ); $session // croak "SNMP error: $error"; my $response = $session->get_request( -varbindlist => $mibs, # MIBΛෳ਺ࢦఆ ) || croak "SNMP error: $session->error";
  70. Nginx HttpStubStatusModule HTTPͰNginxݻ༗ͷϝτϦΫεΛͱΕΔ Active connections: 291 server accepts handled requests

    16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 $ curl http://nginxhost/nginx_status:8080
  71. Plack Plack::Middleware::ServerStatus::Lite JSON format͕͋ΔͷͰָ % curl http://server:port/server-status?json {"Uptime":"1332476669","BusyWorkers":"2", "stats":[ {"protocol":null,"remote_addr":null,"pid":"78639",

    "status":".","method":null,"uri":null,"host":null,"ss":null}, {"protocol":"HTTP/ 1.1","remote_addr":"127.0.0.1","pid":"78640", "status":"A","method":"GET","uri":"/","host":"localhost: 10226","ss":0}, ... ],"IdleWorkers":"3"}
  72. Memcached TelnetΠϯλϑΣʔε stats STAT pid 14868 STAT uptime 175931 STAT

    time 1220540125 STAT version 1.2.2 ... STAT curr_connections 92 STAT total_connections 1740 STAT connection_structures 165 STAT cmd_get 7411 STAT cmd_set 28445156 STAT get_hits 5183 STAT get_misses 2228 STAT evictions 0 STAT bytes_read 2112768087 STAT bytes_written 1000038245 STAT limit_maxbytes 52428800 STAT threads 1 END my $sock = IO::Socket::INET->new( PeerAddr => "$hostname:$port", Proto => 'tcp', Timeout => 10, ) or croak "Couldn't connect to $hostname:$port"; $sock->print("stats\r\n"); my $value_by_stat = {}; while (my $line = $sock->getline) { last if $line =~ /^END/; $line =~ s/\n$|\r\n$//; #chomp if ($line =~ /^STAT\s+(\S*)\s+(.*)/) { $value_by_stat->{$1} = $2; } }
  73. Pullܕ (࠶ܝ) Host Host Host Host Process Storage ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ ετϨʔδʹ֨ೲ

    ετϨʔδ͔Β஋ΛऔΓग़͢
  74. Process

  75. Process ϗετ͸ͨ͘͞Μ͋ΔͷͰฒྻ(ඇಉظ)Ͱɹɹɹ ϝτϦΫεऔಘ͍ͨ͠ ͔͠͠ɺϗετ਺෼ͷϓϩηεΛཱͯΔͷ͸ ͪΐͬͱ... PerlͰεϨου...

  76. WorkerManager Job Queue(The Schwartz)ʹରԠͨ͠ϓϩηε؅ཧ σʔϞϯ Preforkܕ: ϓϩηε਺Λ੍ݶͰ͖Δ ͔ΘΓʹParallel::Prefork΍Starlet/Starman Λ࢖ͬͯ΋Α͍

  77. Jobͷॲཧ಺༰

  78. Agent ֤JobΛAgentͱݺΜͰ͍Δ ϦϞʔτϗετ͔ΒϝτϦΫεΛͱͬͯ͘Δϓϩηε ରԠϛυϧ΢ΣΞɼϓϩτίϧ Apache, Nginx, MySQL, Munin, Latency, Plack,

    Perlbal, Redis, SNMP, Solr, TheSchwartz ϓϥΨϒϧ Hatena::Mackerel::Worker::Agent::XXX SNMP΍HTTP, TelnetͰϝτϦΫεऔಘ
  79. ϓϥάΠϯ͸PerlͰͳ͍ͱॻ͚ͳ͍ ݴޠඇґଘͰϓϥάΠϯΛॻ͖͍ͨ

  80. ͦΕMunin͡ΌͶ MuninͷϓϥάΠϯ͸ ඪ४ೖग़ྗͰύϥϝʔλͷड͚౉͠Λ͢Δ

  81. munin-nodeͱͷ࿈ܞ munin-nodeΛϦϞʔτϗετʹཱ͓ͯͯ͘ mackerel agent͕TelnetͰmunin pluginͷ݁Ռऔಘ munin-node munin plugin mackerel agent

    telnet ඪ४ೖग़ྗ
  82. munin-nodeͱซ༻ ϛυϧ΢ΣΞ͕ϝτϦΫεऔಘͷͨΊͷωοτϫʔ ΫΠϯλϑΣʔε͕͋ΔͳΒ͹ mackerel agent͔ΒωοτϫʔΫɾΠϯλ ϑΣʔεΛ࢖͏ netstat΍smartctlͳͲϦϞʔτϗετͷίϚϯ υΛୟ͖͍ͨͳΒ͹ munin-nodeΛ࢖͏

  83. ϝτϦΫεͷߋ৽ (ετϨʔδͷ࿩)

  84. Pullܕ (࠶ܝ) Host Host Host Host Worker Manager Storage

  85. Storage RDBMS, KVS, other NOSQL...

  86. RRDtool

  87. RRD (Round Robin Database) ࣌ܥྻσʔλΛ֨ೲ͢ΔͨΊͷσʔλϑΥʔϚοτ อଘظؒΛݻఆͯ͠ݹ͍஋͸ࣺ͍ͯͯ͘ RRDtool RRDͷ࣮૷ʢσϑΝΫτελϯμʔυʣ RRDͷ࡞੒ɾߋ৽ɾࢀরػೳͱάϥϑԽػೳΛ΋ͭ I/O͕΍ͨΒ଎͍(Β͍͠)

    େ఍ͷαʔό؅ཧπʔϧͰ࢖ΘΕ͍ͯΔ ͳ͔ͥάϥϑը૾ੜ੒ػೳ͕͋Δ
  88. RRDtoolϝϦοτɾσϝϦοτ ϝϦοτ ࣌ܥྻσʔλΛ͍͍͔Μ͡ʹॻ͖ࠐΜͰ͘ΕΔ ߋ৽ִ͔ؒΒ࣮ࡍͷॻ͖ࠐΈ࣌ࠁ͕গʑͣΕͯ΋मਖ਼ͯ͘͠ ΕΔ (10:01 -> 10:00) ஋ͷਫ਼౓ͷௐઅ͕Մೳ ϥ΢ϯυϩϏϯͳͷͰϑΝΠϧαΠζ͕Ұఆ

    I/O͕ߴ଎ σϝϦοτ rrdtoolʹ༩͑Δύϥϝʔλͷઃఆ͕൥ࡶ
  89. ొ৔ਓ෺͕ͰͦΖͬͨͷͰ γεςϜશମΛ੔ཧ

  90. Job Queue DB RRDtool Host Host Host Host enqueue.pl cron

    δϣϒ౤ೖ ϗετ৘ใऔಘ ϝτϦΫεऔಘ
  91. RRDtool App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ

  92. RRDtool App Server Browser HTTP graph response graph image άϥϑϨεϙϯε

  93. Job Queue DB RRD Host Host Host Host enqueue.pl cron

    RRDtool App Server Browser γεςϜશମ
  94. Web͔Βάϥϑੜ੒ͤ͞Δ ࢓૊Έ

  95. RRD App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ

    ʢ࠶ܝʣ
  96. RRD App Server Browser HTTP graph request rrdtool graph άϥϑϦΫΤετ

    ʢ࠶ܝʣ άϥϑੜ੒URLΛͲ͏දݱ ͢Δ͔
  97. ཁ͸rrdtool graphίϚϯυͷҾ਺ ΛURLͰͲ͏දݱ͢Δ͔

  98. rrdtool graph $ rrdtool graph --end now --start end-120000s --width

    400 \\ DEF:ds0a=/home/rrdtool/data/router1.rrd:ds0:AVERAGE \\ DEF:ds0b=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=1800 \\ DEF:ds0c=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=7200 \\ LINE1:ds0a#0000FF:"default resolution\l" \\ LINE1:ds0b#00CCFF:"resolution 1800 seconds per interval\l" \\ LINE1:ds0c#FF00FF:"resolution 7200 seconds per interval\l" άϥϑඳըظؒͷࢦఆ άϥϑ෯ࢦఆ σʔλϦιʔεʢRRDϑΝΠϧ໊ʣࢦఆ άϥϑઢͷଠ͞΍৭ɺຌྫΛࢦఆ ...
  99. γϯϓϧURL ͱ RRDtool URL ͷ2छྨ༻ҙ

  100. γϯϓϧURL /graph? define=tesuhost01::_default::loadavg5::STACK,testhost0 2::mysql::second_behind_master define=<host>::<tag>::<metrics name>, define=<host>::<tag>::<metrics name>, ... σϑΥϧτ஋ΛܾΊͯɺͳΔ΂͘γϯϓϧʹ

    ઢͷଠ͞ͱ͔͸σϑΥϧτ஋Λ಺෦ͰܾΊ͓ͯ͘
  101. RRDtool URL RRDtoolίϚϯυΛͦͷ··URDͰදݱ hit rateͷܭࢉͱ͔Λͤ͞Δͱ͖ʹ࢖͏ /graph/general?s=[ (def:cmd_get:::=path:memcachedhost,memcached,cmd_get:value:AVERAGE), (def:get_hits:::=path:memcachedhost,memcached,get_hits:value:AVERAGE), (cdef:hit_rate:::=get_hits,cmd_get,/), (line1:hit_rate:::@0000ff:memcachedhost-memcached-hit_rate)

    !end=now-0d,height=100,start=now-30d,width=400 ] memcachedͷhit rateάϥϑURL
  102. my $fmt = <<'EOS'; graphformula = "[graph:" ( graphs [

    ":" option ] ) "]" graphs = graph | "(" graph ")" 0*( ",(" graph ")" ) graph = "path:" elements | instruction ":" [ name ] ":::" option ) elements = object : [ tag : [ label ] ] object = name tag = name label = name instruction = "def" | "cdef" | "vdef" | "line" digit | "area" | "hrule" | "vrule" | "print" | "gprint" | "commend" | "tick" | "shift" | "textalign" option = 1*( char | "," | "=" | "#" | "@" | ":" | " " | "\" | op) name = 1*char char = alphanum | mark op = "+" | "/" | "%" | "*" mark = "-" | "_" | "." | "!" | "~" | "*" | "'" alphanum = alpha | digit alpha = lowalpha | upalpha lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" upalpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" | "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" | "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z" digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" EOS BNFͰਅ໘໨ʹ࢓༷Λఆٛ
  103. ύϑΥʔϚϯεվળ

  104. RRDͷσΟεΫI/Oෛՙ (ϗετ਺) x (ϝτϦΫεछผ਺) io per 5෼ ϗετ਺ = ਺ઍ

    ϝτϦΫεछผ਺ = 20Ҏ্ 10k io per 5෼ ࣮ࡍ͸5෼ͷ͏ͪͷ͋Δ࣌ؒʹI/O͕ूத
  105. 2छྨͷෛՙ࡟ݮํ๏

  106. rrdcached rrdtoolͷߋ৽΍ू໿ॲཧΛड͚෇͚ΔσʔϞϯ rrdͷߋ৽͸ಉ࣌͡ࠁʹࡴ౸͕ͪ͠ ॻ͖ࠐΈΛϥϯμϜͳ࣌ؒ෼஗Ԇͯ͘͠ΕΔ rrdcachedͳͩ͠ͱ, ฏۉ 700 iops όʔετ஋ 1200

    iops SSDͩͱͳΜͱ͔ͳΔ HDD(AWS)ͩͱݫ͍͠ ϦϦʔε൛(1.4.8)͸updateʹ͔͠ରԠ͍ͯ͠ͳ͍ ͕trunkͳΒ͹ͦͷଞͷ֤छαϒίϚϯυʹରԠ
  107. ෼ࢄRRD RRDͷσΟεΫI/OΛϦϞʔτϗετ୯ҐͰ෼ࢄ ͋Δϗετ͕ͲͷRRDϗετʹׂΓ౰ͯΒΕ͍ͯΔ ͔ͷ৘ใΛDBʹ͓࣋ͬͯ͘ Job Queue DB RRD enqueue.pl Job

    Queue RRD Job Queue RRD
  108. Future Work

  109. mackereld

  110. mackereld ϝτϦΫεऔಘͷͨΊͷσʔϞϯ munin-node૬౰ͷػೳ REST APIͰϝτϦΫεऔಘ ϦϞʔτϗετͷϩάΛWebͰϦΞϧλΠϜɹɹ ʹΈΕΔ ֤छϩάϑΝΠϧΛ؂ࢹ͠ɺมߋΛݕ஌ͨ͠Β Server-Sent EventsͰͱ͹͢

    ܧଓతͳมߋ͕΍͸ΓΊΜͲ͏
  111. RRDtoolʹΑΔ༧ଌάϥϑ

  112. RRDtoolʹΑΔ༧ଌάϥϑ σΟεΫ͕ຬഋʹͳΔͷ͸͍ͭʁ 2014೥4݄

  113. $ rrdtool graph ./dskUsed.png \ --start="now-1y" \ --end="now+2y" \ --imgformat=PNG

    \ --title="Disk Usage" \ --height=200 \ --width=500 \ --lower-limit=0 \ DEF:usage=testhost__dskUsed.rrd:value:MAX \ DEF:total=testhost__dskTotal.rrd:value:MAX \ CDEF:c_warn=total,0.85,* \ CDEF:c_crit=total,0.95,* \ VDEF:v_total=total,MINIMUM \ VDEF:v_warn=c_warn,MINIMUM \ VDEF:v_crit=c_crit,MINIMUM \ VDEF:v_usage_slope=usage,LSLSLOPE \ VDEF:v_usage_intercept=usage,LSLINT \ CDEF:c_usage_predict=usage,POP,v_usage_slope,COUNT,*,v_usage_intercept,+ \ CDEF:c_rwarn=c_usage_predict,v_warn,v_total,LIMIT \ VDEF:v_rwarn=c_rwarn,FIRST \ CDEF:c_rcrit=c_usage_predict,v_crit,v_total,LIMIT \ VDEF:v_rcrit=c_rcrit,FIRST \ HRULE:v_warn#FF8800:"warning":dashes=5 \ VRULE:v_rwarn#FF8800::dashes=5 \ HRULE:v_crit#FF4400:"critical" \ VRULE:v_rcrit#FF4400 \ HRULE:v_total#FF0000:"total" \ AREA:usage#00FF00:"Disk Usage" \ LINE1:c_usage_predict#0000FF:"Predict" \ GPRINT:v_rwarn:"Reach warning (85%)\: %c":strftime \ GPRINT:v_rcrit:"Reach critical (95%)\: %c":strftime shoichimasuhara++
  114. Rich Visualization

  115. Rich Visualization ত࿨ͬΆ͍(ը૾) ฏ੒ੜ·Ε͸ͬͪ͜(SVG) RRDtool D3.js

  116. D3.js NVD3.js (Re-usable charts for d3.js) Rickshaw (time-series graph) rrdtool

    xport --json ͰJSONܗࣜͰ஋Λऔಘɹɹ Ͱ͖ΔͨΊɺετϨʔδ͸RRDͷ·· D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS.
  117. RRDtool͔Βͷ୤٫

  118. ୅ΘΓʹRedis΍MongoDBͳͲͷετϨʔδΛ࢖͏ FnordMetrics (http://fnordmetric.io/) GrowthForecastʹࣅͯΔ ετϨʔδɿRedis άϥϑɿRickshaw my $furl = Furl->new(timeout

    => 10); my $res = $furl->post( "http://fnordmetricshost:4242/events", ['Content-Type' => 'application/json'], encode_json({ "_type" => '_incr', "value" => scalar @$statuses, "gauge" => "tweets_per_minute" }), );
  119. αʔόϝτϦΫεՄࢹԽͷ࿩ ͓ΘΓ

  120. ·ͱΊ

  121. ϗετ৘ใͷҰݩ؅ཧ ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍ ֤छπʔϧͱૄ݁߹ʹ౷߹ REST APIʹΑΔपลπʔϧͱͷ࿈ܞ

  122. ϝτϦΫεՄࢹԽͷ࣮૷ PullܕΞʔΩςΫνϟ WorkerManagerͰϓϩηε؅ཧ Storageʹ͸RRDtoolΛ࢖͏ RRDtool΂ΜΓ RRDtool͍ͨ΁Μ RRDtool͸΍͍ RRDtoolΉ͔͍ͣ͠

  123. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ Thanks to id:stanaka, id:halfrack, id:wtatsuru, id:shoichimasuhara, id:hagihala