はてなのサーバ管理ツールの話

はてなのサーバ管理ツールの思想とサーバメトリクス可視化システムの実装

Transcript

1. ͸ͯͳͷαʔό؅ཧπʔϧͷ࿩
   Mackerel - Server Management with Tool
   YAPC::Asia Tokyo 2013
   

   View Slide

2. HatenaJEZ@VVLJ
   Twitter!Z@VVL
   CPAN:66,*
   ͸ͯͳࣾΞϧόΠτࢯ
   !γεςϜϓϥοτϑΥʔϜ෦
   

   View Slide

3. ࠓ೔ͷ༷ࢠͰ͢
   

   View Slide

4. ͍͖͞Μ৬࣭͞Ε·ͨ͠
   http://moznion.hatenadiary.com/entry/2013/08/14/010104
   id:moznionࢯ
   

   View Slide

5. Perlྺ1೥͘Β͍
   

   View Slide

6. Perl
   CPAN
   WebService::Qiita
   RRDTool::Rawish
   Plack::Middleware::GitStatus
   Net::Signalet
   vim-plugin
   unite-perl-module.vim
   perl-local-lib-path.vim
   

   View Slide

7. Kyoto.pm
   

   View Slide

8. YAPCࢀՃ͸͡ΊͯͰ͢
   

   View Slide

9. ͻʔ
   y_uuki
   

   View Slide

10. 40෼͠Ό΂Γ·͢
    

    View Slide

11. ͦͷ·͑ʹ
    

    View Slide

12. ӡ༻ܥͷํʁ
    

    View Slide

13. Muninͱ͔Zabbixܥͷπʔϧ
    Λ࢖ͬͯΒͬ͠ΌΔํʁ
    

    View Slide

14. RRDtoolΛ͝ଘ஌ͷํ?
    

    View Slide

15. ࠓ೔ͷ͓࿩
    ͸ͯͳͷαʔό؅ཧπʔϧͷࢥ૝
    ύϑΥʔϚϯεϝτϦΫε
    ՄࢹԽͷ࢓૊Έͷ࣮૷
    

    View Slide

16. αʔό؅ཧπʔϧ
    Munin Cacti Zabbix Ganglia collectd etc...
    

    View Slide

17. ͍Ζ͍Ζ͋Δ
    

    View Slide

18. αʔό؅ཧपลπʔϧ
    ؂ࢹܥ
    ߏ੒؅ཧܥ
    σϓϩΠܥ
    Cinnamon
    

    View Slide

19. ͍Ζ͍Ζ͋Δ
    

    View Slide

20. ໰୊఺
    

    View Slide

21. ϗετ৘ใ͕෼ࢄͯ͠͠·͏
    ໨తผʹπʔϧ͕ଘࡏ͢Δ
    Nagiosͷconfig
    Capistranoͷconfig.rb
    Muninͷconfig
    Chef Server
    

    View Slide

22. ͡Ό͋ϑϧελοΫܥͷ
    πʔϧ࢖͑͹?
    

    View Slide

23. ϑϧελοΫπʔϧ
    Zabbix (ϗετ؅ཧ + ؂ࢹ)
    Crowbar (Ganglia + Chef + Nagios)
    ࢖͍͍ͨπʔϧͱ૊Έ߹Θͤʹ͍͘
    ֤ػೳ͕ີ݁߹
    

    View Slide

24. ϗετ৘ใΛ෼ࢄͤͣ͞ҰݩԽ͍ͨ͠
    ֤πʔϧͱૄ݁߹ʹ౷߹͍ͨ͠
    ࣗ࡞͠Α͏ʂ
    

    View Slide

25. ϑϧεΫϥον
    http://anond.hatelabo.jp/20120313004820
    

    View Slide

26. Mackerel
    ʲՄࢉ໊ࢺʳɹαό
    ʲෆՄࢉ໊ࢺʳαό(ͷ਎)
    Plack / Starlet / Teng / Router::Simple /
    Class::Accessor::Lite::Lazy / TheSchwartz / Text::Xslate /
    Config::ENV / Scope::Container / SQL::Maker ...
    Perl 5.14 / Carton 1.0
    

    View Slide

27. ·ͣ͸ࢥ૝͔Β
    

    View Slide

28. ϗετ৘ใͷҰݩ؅ཧ
    ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍
    ֤छπʔϧͱૄ݁߹ʹ౷߹
    REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
    

    View Slide

29. ϗετ৘ใͷҰݩ؅ཧ
    ϗετ؅ཧ
    ෺ཧϗετ or Ծ૝ϗετ or AWS
    ෺ཧϗετͱԾ૝ϗετͷରԠؔ܎
    ͲͷαʔϏεͷͲ͏͍͏໾ׂ(DBαʔό,ϓϩΩγαʔόͳͲʣ
    ͲͷσʔληϯλͷͲͷϥοΫ
    ͲͷωοτϫʔΫʢαϒωοτʣ
    ঢ়ଶ
    (working,standby,maintenace,poweroff,destroyed)
    

    View Slide

30. ServiceҰཡ
    

    View Slide

31. ServiceҎԼͷRoleҰཡ
    RoleҰཡ
    Role
    ϗετ৘ใ
    

    View Slide

32. ৄࡉϗετ৘ใ
    load average CPU time
    iops memory usaage
    ... other graphs
    

    View Slide

33. पลπʔϧͱͷ࿈ܞ
    

    View Slide

34. Nagios࿈ܞ
    

    View Slide

35. ઃఆͷࣗಈੜ੒
    MackerelͷAPI͔Βϗετ৘ใΛऔಘͯ͠ϗετఆٛ
    ϑΝΠϧΛࣗಈੜ੒
    NagiosͷϗετstatusΛMackerelͷϏϡʔʹ΋දࣔ
    NagiosͷWeb UIʹͱ͹ͳ͍͍ͯ͘Α͏ʹ
    ৄ͘͠͸ίνϥ
    ϞχΧδ#3 http://www.slideshare.net/shoichimasuhara/3-ss-17097401
    Nagios࿈ܞ
    

    View Slide

36. Chef࿈ܞ
    (Just Idea)
    

    View Slide

37. Chef Serverͱͷಉظ
    MackerelͷDBΛϚελͱ͢Δ
    Mackerelʹߋ৽ɾ࡟আ͕૸Ε͹Chef ServerͷAPIΛୟ
    ͍ͯPOST
    Chef Serverͷഇࢭ
    MackerelͷRole͝ͱʹϨγϐΛ༻ҙ
    Ϩγϐͷద༻ઌ͸MackerelͷAPI͔Β౎౓औಘ
    Knife SoloͰϨγϐద༻
    

    View Slide

38. Capistrano࿈ܞ
    

    View Slide

39. Capistrano࿈ܞ
    Capfileʹ௚ʹσϓϩΠઌϦετΛॻ͖ͨ͘ͳ͍
    σϓϩΠ͢ΔͨͼʹMackerelͷAPIΛୟ͍ͯσϓϩΠઌ
    ϗετҰཡΛऔಘ
    http://mackerelhost/service/Hatena-Blog/roles/
    backend?format=json
    backend = ΞϓϦέʔγϣϯαʔό
    ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷΑ͏ʹRoleͰࢦఆ
    

    View Slide

40. IRC࿈ܞ
    

    View Slide

41. IRC࿈ܞ
    ϗετ৘ใͷ࡞੒ɾߋ৽ɾ࡟আΛIkachanܦ༝Ͱ
    IRCʹ௨஌
    ୭͕ͲͷΑ͏ͳૢ࡞Λ͔͕ͨ͠ϦΞϧλΠϜʹΘ͔Δ
    DBૢ࡞ʹରͯ͠௨஌ͷHookΛ࢓ࠐΉ
    (wtatsuru) update host 'intern2013': {"status"=>"maintenance"}
    εςʔλεΛ”ϝϯςφεத”ʹҠߦ
    

    View Slide

42. DNS࿈ܞ
    

    View Slide

43. RoleΛߏ੒͢Δϗετ
    reverse
    proxy LVS
    Role Role
    ৑௕Խɾෛՙ෼ࢄ͞Ε͍ͯΔ͜ͱʹண໨
    

    View Slide

44. ৑௕Խɾෛՙ෼ࢄ
    Role
    Ծ૝IP(VIP)
    LVS
    Role
    DNS
    ϥ΢ϯυϩϏϯ
    DNS
    DNSϥ΢ϯυϩϏϯ༻FQDNʹର
    ͯ͠ෳ਺ͷIP͔Βϥ΢ϯυϩϏ
    ϯͰҰͭʹIPΛฦ͢
    LVSϗετʹԾ૝IPΛׂΓ౰ͯ
    Ծ૝IP͑͞Θ͔͍ͬͯΕ͹ɹɹ
    ͋ͱ͸উखʹ෼ࢄ
    

    View Slide

45. ϥ΢ϯυϩϏϯ༻FQDNͱ
    VIP͸͋·Γม͑ͨ͘ͳ͍
    Service໊ͱRole໊͸΄ͱΜͲมΘΒ
    ͳ͍͜ͱʹண໨
    

    View Slide

46. DNSϥ΢ϯυϩϏϯ
    DNSϥ΢ϯυϩϏϯ༻ͷFQDNΛService-Roleϕʔε
    ͷ໊લͰׂΓ౰ͯ
    nslookup hosts.db-slave.bookmark
    Name: hosts.db-slave.bookmark
    Address: 10.0.0.1
    Name: hosts.db-slave.bookmark
    Address: 10.0.0.2
    Name: hosts.db-slave.bookmark
    Address: 10.0.0.3
    role.service
    

    View Slide

47. VIP
    LVSͷVIPʹରͯ͠FQDNΛService-Roleϕʔε
    ͷ໊લͰׂΓ౰ͯ
    nslookup db-slave.bookmark
    Name: db-slave.bookmark
    Address: 192.168.1.1
    role.service
    

    View Slide

48. पลπʔϧͱͷ࿈ܞͷ࿩ऴΘΓ
    

    View Slide

49. αʔό؅ཧʹ޲͍ͨϏϡʔ
    

    View Slide

50. ৘ใີ౓͕ߴ͘໢ཏతͳϏϡʔ
    ڝഅ৽ฉΈ͍ͨͳΠϝʔδ
    ۭ͖༰ྔͱ͔͕ͬ͟ͱΈΕΔ
    

    View Slide

51. ϗετ໊/IP
    ͳͲ
    loadavg
    άϥϑ
    Datacenter/
    Rack
    Service/
    Role
    Resource Memo
    

    View Slide

52. ͍ͭ΋ΈΔάϥϑ΍ൺֱ͍ͨ͠άϥϑΛબΜͰ̍ϖʔδ
    ʹ·ͱΊΔ
    HTMLʹՃ͑ͯɺάϥϑه๏Λ࢖ͬͯΧελϚΠζ
    
    
    
    [:graph:service=SERVICE:role=ROLE]
    [:graph:service=SERVICE:role=ROLE]
    
    ΧελϚΠζͰ͖ΔμογϡϘʔυ
    

    View Slide

53. View Slide

54. Roleϗετͷάϥϑൺֱ
    ͸ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷload averageൺֱ
    ҟৗͳϗετ͕͍ͳ͍͔Ͳ͏͔ݕग़
    

    View Slide

55. લ൒͓ΘΓ
    

    View Slide

56. αʔόϝτϦΫεՄࢹԽͷ
    ࢓૊Έ
    ϝτϦΫεMPBEBWFSBHFɼ$16UJNFɼNFNPSZVTBHF FUD
    

    View Slide

57. αʔόϝτϦΫεՄࢹԽͬͯʁ
    

    View Slide

58. View Slide

59. MackerelͰͲ͏΍࣮ͬͯ૷ɹ
    ͯ͠Δ͔ͷ࿩
    

    View Slide

60. αʔόϝτϦΫεՄࢹԽʹඞਢͷཁૉ
    

    View Slide

61. αʔόϝτϦΫεՄࢹԽ
    Host
    Host
    Host
    Host
    Process
    Storage
    ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
    ετϨʔδʹ֨ೲ
    ετϨʔδ͔Β஋ΛऔΓग़͢
    άϥϑԽ
    

    View Slide

62. ͲͷΑ͏ͳΞʔΩςΫνϟ͕Α͍͔
    

    View Slide

63. Push or Pull
    

    View Slide

64. Pushܕ
    Host
    Host
    Host
    Host
    Process
    Storage
    ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
    Host͔ΒProcess΁
    ϝτϦΫεΛPush
    

    View Slide

65. Pushܕͷ໰୊఺
    ֤ϗετʹcronͰճ͢ϝτϦΫεऔಘ&ૹ৴εΫ
    ϦϓτΛ഑෍͠ͳ͚Ε͹ͳΒͳ͍
    εΫϦϓτͷܧଓతͳมߋ͕໘౗
    

    View Slide

66. Pullܕ
    Host
    Host
    Host
    Host
    Process
    Storage
    ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
    Process͔ΒHost΁
    ϦΫΤετ͢Δ
    

    View Slide

67. PullܕͰ΋݁ہࣗલͷ
    ʢܧଓతͳมߋ͕ඞཁͳʣσʔϞϯΛ
    ཱͯͳ͚Ε͹ͳΒͳ͍ͷͰ͸ʁ
    

    View Slide

68. ϝτϦΫεऔಘͷͨΊͷ
    ωοτϫʔΫΠϯλϑΣʔεΛ΋ͭ
    طଘͷσʔϞϯΛ࢖͏
    

    View Slide

69. SNMP
    OS͕ఏڙ͢ΔجຊతͳϝτϦΫεΛऔಘ
    ͦͷଞϧʔλ΍SquidͳͲ
    Net::SNMPΛ࢖͏
    my ($session, $error) = Net::SNMP->session(
    -hostname => $hostname,
    -community => $community,
    -version => 2,
    -timeout => 10,
    -translate => 0x0,
    );
    $session // croak "SNMP error: $error";
    my $response = $session->get_request(
    -varbindlist => $mibs, # MIBΛෳ਺ࢦఆ
    ) || croak "SNMP error: $session->error";
    

    View Slide

70. Nginx
    HttpStubStatusModule
    HTTPͰNginxݻ༗ͷϝτϦΫεΛͱΕΔ
    Active connections: 291
    server accepts handled requests
    16630948 16630948 31070465
    Reading: 6 Writing: 179 Waiting: 106
    $ curl http://nginxhost/nginx_status:8080
    

    View Slide

71. Plack
    Plack::Middleware::ServerStatus::Lite
    JSON format͕͋ΔͷͰָ
    % curl http://server:port/server-status?json
    {"Uptime":"1332476669","BusyWorkers":"2",
    "stats":[
    {"protocol":null,"remote_addr":null,"pid":"78639",
    "status":".","method":null,"uri":null,"host":null,"ss":null},
    {"protocol":"HTTP/
    1.1","remote_addr":"127.0.0.1","pid":"78640",
    "status":"A","method":"GET","uri":"/","host":"localhost:
    10226","ss":0},
    ...
    ],"IdleWorkers":"3"}
    

    View Slide

72. Memcached
    TelnetΠϯλϑΣʔε
    stats
    STAT pid 14868
    STAT uptime 175931
    STAT time 1220540125
    STAT version 1.2.2
    ...
    STAT curr_connections 92
    STAT total_connections 1740
    STAT connection_structures 165
    STAT cmd_get 7411
    STAT cmd_set 28445156
    STAT get_hits 5183
    STAT get_misses 2228
    STAT evictions 0
    STAT bytes_read 2112768087
    STAT bytes_written 1000038245
    STAT limit_maxbytes 52428800
    STAT threads 1
    END
    my $sock = IO::Socket::INET->new(
    PeerAddr => "$hostname:$port",
    Proto => 'tcp',
    Timeout => 10,
    ) or croak "Couldn't connect to $hostname:$port";
    $sock->print("stats\r\n");
    my $value_by_stat = {};
    while (my $line = $sock->getline) {
    last if $line =~ /^END/;
    $line =~ s/\n$|\r\n$//; #chomp
    if ($line =~ /^STAT\s+(\S*)\s+(.*)/) {
    $value_by_stat->{$1} = $2;
    }
    }
    

    View Slide

73. Pullܕ (࠶ܝ)
    Host
    Host
    Host
    Host
    Process
    Storage
    ֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
    ετϨʔδʹ֨ೲ
    ετϨʔδ͔Β஋ΛऔΓग़͢
    

    View Slide

74. Process
    

    View Slide

75. Process
    ϗετ͸ͨ͘͞Μ͋ΔͷͰฒྻ(ඇಉظ)Ͱɹɹɹ
    ϝτϦΫεऔಘ͍ͨ͠
    ͔͠͠ɺϗετ਺෼ͷϓϩηεΛཱͯΔͷ͸
    ͪΐͬͱ...
    PerlͰεϨου...
    

    View Slide

76. WorkerManager
    Job Queue(The Schwartz)ʹରԠͨ͠ϓϩηε؅ཧ
    σʔϞϯ
    Preforkܕ: ϓϩηε਺Λ੍ݶͰ͖Δ
    ͔ΘΓʹParallel::Prefork΍Starlet/Starman
    Λ࢖ͬͯ΋Α͍
    

    View Slide

77. Jobͷॲཧ಺༰
    

    View Slide

78. Agent
    ֤JobΛAgentͱݺΜͰ͍Δ
    ϦϞʔτϗετ͔ΒϝτϦΫεΛͱͬͯ͘Δϓϩηε
    ରԠϛυϧ΢ΣΞɼϓϩτίϧ
    Apache, Nginx, MySQL, Munin, Latency,
    Plack, Perlbal, Redis, SNMP, Solr,
    TheSchwartz
    ϓϥΨϒϧ
    Hatena::Mackerel::Worker::Agent::XXX
    SNMP΍HTTP, TelnetͰϝτϦΫεऔಘ
    

    View Slide

79. ϓϥάΠϯ͸PerlͰͳ͍ͱॻ͚ͳ͍
    ݴޠඇґଘͰϓϥάΠϯΛॻ͖͍ͨ
    

    View Slide

80. ͦΕMunin͡ΌͶ
    MuninͷϓϥάΠϯ͸
    ඪ४ೖग़ྗͰύϥϝʔλͷड͚౉͠Λ͢Δ
    

    View Slide

81. munin-nodeͱͷ࿈ܞ
    munin-nodeΛϦϞʔτϗετʹཱ͓ͯͯ͘
    mackerel agent͕TelnetͰmunin pluginͷ݁Ռऔಘ
    munin-node
    munin plugin
    mackerel
    agent
    telnet
    ඪ४ೖग़ྗ
    

    View Slide

82. munin-nodeͱซ༻
    ϛυϧ΢ΣΞ͕ϝτϦΫεऔಘͷͨΊͷωοτϫʔ
    ΫΠϯλϑΣʔε͕͋ΔͳΒ͹
    mackerel agent͔ΒωοτϫʔΫɾΠϯλ
    ϑΣʔεΛ࢖͏
    netstat΍smartctlͳͲϦϞʔτϗετͷίϚϯ
    υΛୟ͖͍ͨͳΒ͹
    munin-nodeΛ࢖͏
    

    View Slide

83. ϝτϦΫεͷߋ৽
    (ετϨʔδͷ࿩)
    

    View Slide

84. Pullܕ (࠶ܝ)
    Host
    Host
    Host
    Host
    Worker
    Manager
    Storage
    

    View Slide

85. Storage
    RDBMS, KVS, other NOSQL...
    

    View Slide

86. RRDtool
    

    View Slide

87. RRD (Round Robin Database)
    ࣌ܥྻσʔλΛ֨ೲ͢ΔͨΊͷσʔλϑΥʔϚοτ
    อଘظؒΛݻఆͯ͠ݹ͍஋͸ࣺ͍ͯͯ͘
    RRDtool
    RRDͷ࣮૷ʢσϑΝΫτελϯμʔυʣ
    RRDͷ࡞੒ɾߋ৽ɾࢀরػೳͱάϥϑԽػೳΛ΋ͭ
    I/O͕΍ͨΒ଎͍(Β͍͠)
    େ఍ͷαʔό؅ཧπʔϧͰ࢖ΘΕ͍ͯΔ
    ͳ͔ͥάϥϑը૾ੜ੒ػೳ͕͋Δ
    

    View Slide

88. RRDtoolϝϦοτɾσϝϦοτ
    ϝϦοτ
    ࣌ܥྻσʔλΛ͍͍͔Μ͡ʹॻ͖ࠐΜͰ͘ΕΔ
    ߋ৽ִ͔ؒΒ࣮ࡍͷॻ͖ࠐΈ࣌ࠁ͕গʑͣΕͯ΋मਖ਼ͯ͘͠
    ΕΔ (10:01 -> 10:00)
    ஋ͷਫ਼౓ͷௐઅ͕Մೳ
    ϥ΢ϯυϩϏϯͳͷͰϑΝΠϧαΠζ͕Ұఆ
    I/O͕ߴ଎
    σϝϦοτ
    rrdtoolʹ༩͑Δύϥϝʔλͷઃఆ͕൥ࡶ
    

    View Slide

89. ొ৔ਓ෺͕ͰͦΖͬͨͷͰ
    γεςϜશମΛ੔ཧ
    

    View Slide

90. Job Queue
    DB
    RRDtool
    Host
    Host
    Host
    Host
    enqueue.pl
    cron
    δϣϒ౤ೖ
    ϗετ৘ใऔಘ
    ϝτϦΫεऔಘ
    

    View Slide

91. RRDtool
    App Server
    Browser
    HTTP graph
    request
    rrdtool graph
    άϥϑϦΫΤετ
    

    View Slide

92. RRDtool
    App Server
    Browser
    HTTP graph
    response
    graph image
    άϥϑϨεϙϯε
    

    View Slide

93. Job Queue
    DB
    RRD
    Host
    Host
    Host
    Host
    enqueue.pl
    cron
    RRDtool
    App Server
    Browser
    γεςϜશମ
    

    View Slide

94. Web͔Βάϥϑੜ੒ͤ͞Δ
    ࢓૊Έ
    

    View Slide

95. RRD
    App Server
    Browser
    HTTP graph
    request
    rrdtool graph
    άϥϑϦΫΤετ
    ʢ࠶ܝʣ
    

    View Slide

96. RRD
    App Server
    Browser
    HTTP graph
    request
    rrdtool graph
    άϥϑϦΫΤετ
    ʢ࠶ܝʣ
    άϥϑੜ੒URLΛͲ͏දݱ
    ͢Δ͔
    

    View Slide

97. ཁ͸rrdtool graphίϚϯυͷҾ਺
    ΛURLͰͲ͏දݱ͢Δ͔
    

    View Slide

98. rrdtool graph
    $ rrdtool graph --end now --start end-120000s --width 400 \\
    DEF:ds0a=/home/rrdtool/data/router1.rrd:ds0:AVERAGE \\
    DEF:ds0b=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=1800 \\
    DEF:ds0c=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=7200 \\
    LINE1:ds0a#0000FF:"default resolution\l" \\
    LINE1:ds0b#00CCFF:"resolution 1800 seconds per interval\l" \\
    LINE1:ds0c#FF00FF:"resolution 7200 seconds per interval\l"
    άϥϑඳըظؒͷࢦఆ
    άϥϑ෯ࢦఆ
    σʔλϦιʔεʢRRDϑΝΠϧ໊ʣࢦఆ
    άϥϑઢͷଠ͞΍৭ɺຌྫΛࢦఆ
    ...
    

    View Slide

99. γϯϓϧURL
    ͱ
    RRDtool URL
    ͷ2छྨ༻ҙ
    

    View Slide

100. γϯϓϧURL
     /graph?
     define=tesuhost01::_default::loadavg5::STACK,testhost0
     2::mysql::second_behind_master
     define=::::,
     define=::::,
     ...
     σϑΥϧτ஋ΛܾΊͯɺͳΔ΂͘γϯϓϧʹ
     ઢͷଠ͞ͱ͔͸σϑΥϧτ஋Λ಺෦ͰܾΊ͓ͯ͘
     

     View Slide

101. RRDtool URL
     RRDtoolίϚϯυΛͦͷ··URDͰදݱ
     hit rateͷܭࢉͱ͔Λͤ͞Δͱ͖ʹ࢖͏
     /graph/general?s=[
     (def:cmd_get:::=path:memcachedhost,memcached,cmd_get:value:AVERAGE),
     (def:get_hits:::=path:memcachedhost,memcached,get_hits:value:AVERAGE),
     (cdef:hit_rate:::=get_hits,cmd_get,/),
     (line1:hit_rate:::@0000ff:memcachedhost-memcached-hit_rate)
     !end=now-0d,height=100,start=now-30d,width=400
     ]
     memcachedͷhit rateάϥϑURL
     

     View Slide

102. my $fmt = <graphformula = "[graph:" ( graphs [ ":" option ] ) "]"
     graphs = graph | "(" graph ")" 0*( ",(" graph ")" )
     graph = "path:" elements | instruction ":" [ name ] ":::" option )
     elements = object : [ tag : [ label ] ]
     object = name
     tag = name
     label = name
     instruction = "def" | "cdef" | "vdef" | "line" digit | "area" | "hrule" | "vrule" | "print" |
     "gprint" | "commend" | "tick" | "shift" | "textalign"
     option = 1*( char | "," | "=" | "#" | "@" | ":" | " " | "\" | op)
     name = 1*char
     char = alphanum | mark
     op = "+" | "/" | "%" | "*"
     mark = "-" | "_" | "." | "!" | "~" | "*" | "'"
     alphanum = alpha | digit
     alpha = lowalpha | upalpha
     lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" |
     "j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" |
     "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z"
     upalpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" |
     "J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" |
     "S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"
     digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" |
     "8" | "9"
     EOS
     BNFͰਅ໘໨ʹ࢓༷Λఆٛ
     

     View Slide

103. ύϑΥʔϚϯεվળ
     

     View Slide

104. RRDͷσΟεΫI/Oෛՙ
     (ϗετ਺) x (ϝτϦΫεछผ਺) io per 5෼
     ϗετ਺ = ਺ઍ
     ϝτϦΫεछผ਺ = 20Ҏ্
     10k io per 5෼
     ࣮ࡍ͸5෼ͷ͏ͪͷ͋Δ࣌ؒʹI/O͕ूத
     

     View Slide

105. 2छྨͷෛՙ࡟ݮํ๏
     

     View Slide

106. rrdcached
     rrdtoolͷߋ৽΍ू໿ॲཧΛड͚෇͚ΔσʔϞϯ
     rrdͷߋ৽͸ಉ࣌͡ࠁʹࡴ౸͕ͪ͠
     ॻ͖ࠐΈΛϥϯμϜͳ࣌ؒ෼஗Ԇͯ͘͠ΕΔ
     rrdcachedͳͩ͠ͱ, ฏۉ 700 iops όʔετ஋
     1200 iops
     SSDͩͱͳΜͱ͔ͳΔ HDD(AWS)ͩͱݫ͍͠
     ϦϦʔε൛(1.4.8)͸updateʹ͔͠ରԠ͍ͯ͠ͳ͍
     ͕trunkͳΒ͹ͦͷଞͷ֤छαϒίϚϯυʹରԠ
     

     View Slide

107. ෼ࢄRRD
     RRDͷσΟεΫI/OΛϦϞʔτϗετ୯ҐͰ෼ࢄ
     ͋Δϗετ͕ͲͷRRDϗετʹׂΓ౰ͯΒΕ͍ͯΔ
     ͔ͷ৘ใΛDBʹ͓࣋ͬͯ͘
     Job Queue
     DB
     RRD
     enqueue.pl
     Job Queue RRD
     Job Queue RRD
     

     View Slide

108. Future Work
     

     View Slide

109. mackereld
     

     View Slide

110. mackereld
     ϝτϦΫεऔಘͷͨΊͷσʔϞϯ
     munin-node૬౰ͷػೳ
     REST APIͰϝτϦΫεऔಘ
     ϦϞʔτϗετͷϩάΛWebͰϦΞϧλΠϜɹɹ
     ʹΈΕΔ
     ֤छϩάϑΝΠϧΛ؂ࢹ͠ɺมߋΛݕ஌ͨ͠Β
     Server-Sent EventsͰͱ͹͢
     ܧଓతͳมߋ͕΍͸ΓΊΜͲ͏
     

     View Slide

111. RRDtoolʹΑΔ༧ଌάϥϑ
     

     View Slide

112. RRDtoolʹΑΔ༧ଌάϥϑ
     σΟεΫ͕ຬഋʹͳΔͷ͸͍ͭʁ
     2014೥4݄
     

     View Slide

113. $ rrdtool graph ./dskUsed.png \
     --start="now-1y" \
     --end="now+2y" \
     --imgformat=PNG \
     --title="Disk Usage" \
     --height=200 \
     --width=500 \
     --lower-limit=0 \
     DEF:usage=testhost__dskUsed.rrd:value:MAX \
     DEF:total=testhost__dskTotal.rrd:value:MAX \
     CDEF:c_warn=total,0.85,* \
     CDEF:c_crit=total,0.95,* \
     VDEF:v_total=total,MINIMUM \
     VDEF:v_warn=c_warn,MINIMUM \
     VDEF:v_crit=c_crit,MINIMUM \
     VDEF:v_usage_slope=usage,LSLSLOPE \
     VDEF:v_usage_intercept=usage,LSLINT \
     CDEF:c_usage_predict=usage,POP,v_usage_slope,COUNT,*,v_usage_intercept,+ \
     CDEF:c_rwarn=c_usage_predict,v_warn,v_total,LIMIT \
     VDEF:v_rwarn=c_rwarn,FIRST \
     CDEF:c_rcrit=c_usage_predict,v_crit,v_total,LIMIT \
     VDEF:v_rcrit=c_rcrit,FIRST \
     HRULE:v_warn#FF8800:"warning":dashes=5 \
     VRULE:v_rwarn#FF8800::dashes=5 \
     HRULE:v_crit#FF4400:"critical" \
     VRULE:v_rcrit#FF4400 \
     HRULE:v_total#FF0000:"total" \
     AREA:usage#00FF00:"Disk Usage" \
     LINE1:c_usage_predict#0000FF:"Predict" \
     GPRINT:v_rwarn:"Reach warning (85%)\: %c":strftime \
     GPRINT:v_rcrit:"Reach critical (95%)\: %c":strftime
     shoichimasuhara++
     

     View Slide

114. Rich Visualization
     

     View Slide

115. Rich Visualization
     ত࿨ͬΆ͍(ը૾)
     ฏ੒ੜ·Ε͸ͬͪ͜(SVG)
     RRDtool
     D3.js
     

     View Slide

116. D3.js
     NVD3.js (Re-usable charts for d3.js)
     Rickshaw (time-series graph)
     rrdtool xport --json ͰJSONܗࣜͰ஋Λऔಘɹɹ
     Ͱ͖ΔͨΊɺετϨʔδ͸RRDͷ··
     D3.js is a JavaScript library for manipulating documents based on data.
     D3 helps you bring data to life using HTML, SVG and CSS.
     

     View Slide

117. RRDtool͔Βͷ୤٫
     

     View Slide

118. ୅ΘΓʹRedis΍MongoDBͳͲͷετϨʔδΛ࢖͏
     FnordMetrics (http://fnordmetric.io/)
     GrowthForecastʹࣅͯΔ
     ετϨʔδɿRedis
     άϥϑɿRickshaw
     my $furl = Furl->new(timeout => 10);
     my $res = $furl->post(
     "http://fnordmetricshost:4242/events",
     ['Content-Type' => 'application/json'],
     encode_json({
     "_type" => '_incr',
     "value" => scalar @$statuses,
     "gauge" => "tweets_per_minute"
     }),
     );
     

     View Slide

119. αʔόϝτϦΫεՄࢹԽͷ࿩
     ͓ΘΓ
     

     View Slide

120. ·ͱΊ
     

     View Slide

121. ϗετ৘ใͷҰݩ؅ཧ
     ͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετ৘ใ͕͋Δͷ͸Α͘ͳ͍
     ֤छπʔϧͱૄ݁߹ʹ౷߹
     REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
     

     View Slide

122. ϝτϦΫεՄࢹԽͷ࣮૷
     PullܕΞʔΩςΫνϟ
     WorkerManagerͰϓϩηε؅ཧ
     Storageʹ͸RRDtoolΛ࢖͏
     RRDtool΂ΜΓ
     RRDtool͍ͨ΁Μ
     RRDtool͸΍͍
     RRDtoolΉ͔͍ͣ͠
     

     View Slide

123. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
     Thanks to id:stanaka, id:halfrack, id:wtatsuru, id:shoichimasuhara, id:hagihala
     

     View Slide