Slide 1
Slide 1 text
ͯͳͷαʔόཧπʔϧͷ
Mackerel - Server Management with Tool
YAPC::Asia Tokyo 2013
Slide 2
Slide 2 text
HatenaJEZ@VVLJ
Twitter!Z@VVL
CPAN:66,*
ͯͳࣾΞϧόΠτࢯ
!γεςϜϓϥοτϑΥʔϜ෦
Slide 3
Slide 3 text
ࠓͷ༷ࢠͰ͢
Slide 4
Slide 4 text
͍͖͞Μ৬࣭͞Ε·ͨ͠
http://moznion.hatenadiary.com/entry/2013/08/14/010104
id:moznionࢯ
Slide 5
Slide 5 text
Perlྺ1͘Β͍
Slide 6
Slide 6 text
Perl
CPAN
WebService::Qiita
RRDTool::Rawish
Plack::Middleware::GitStatus
Net::Signalet
vim-plugin
unite-perl-module.vim
perl-local-lib-path.vim
Slide 7
Slide 7 text
Kyoto.pm
Slide 8
Slide 8 text
YAPCࢀՃ͡ΊͯͰ͢
Slide 9
Slide 9 text
ͻʔ
y_uuki
Slide 10
Slide 10 text
40͠ΌΓ·͢
Slide 11
Slide 11 text
ͦͷ·͑ʹ
Slide 12
Slide 12 text
ӡ༻ܥͷํʁ
Slide 13
Slide 13 text
Muninͱ͔Zabbixܥͷπʔϧ
ΛͬͯΒͬ͠ΌΔํʁ
Slide 14
Slide 14 text
RRDtoolΛ͝ଘͷํ?
Slide 15
Slide 15 text
ࠓͷ͓
ͯͳͷαʔόཧπʔϧͷࢥ
ύϑΥʔϚϯεϝτϦΫε
ՄࢹԽͷΈͷ࣮
Slide 16
Slide 16 text
αʔόཧπʔϧ
Munin Cacti Zabbix Ganglia collectd etc...
Slide 17
Slide 17 text
͍Ζ͍Ζ͋Δ
Slide 18
Slide 18 text
αʔόཧपลπʔϧ
ࢹܥ
ߏཧܥ
σϓϩΠܥ
Cinnamon
Slide 19
Slide 19 text
͍Ζ͍Ζ͋Δ
Slide 20
Slide 20 text
Slide 21
Slide 21 text
ϗετใ͕ࢄͯ͠͠·͏
తผʹπʔϧ͕ଘࡏ͢Δ
Nagiosͷconfig
Capistranoͷconfig.rb
Muninͷconfig
Chef Server
Slide 22
Slide 22 text
͡Ό͋ϑϧελοΫܥͷ
πʔϧ͑?
Slide 23
Slide 23 text
ϑϧελοΫπʔϧ
Zabbix (ϗετཧ + ࢹ)
Crowbar (Ganglia + Chef + Nagios)
͍͍ͨπʔϧͱΈ߹Θͤʹ͍͘
֤ػೳ͕ີ݁߹
Slide 24
Slide 24 text
ϗετใΛࢄͤͣ͞ҰݩԽ͍ͨ͠
֤πʔϧͱૄ݁߹ʹ౷߹͍ͨ͠
ࣗ࡞͠Α͏ʂ
Slide 25
Slide 25 text
ϑϧεΫϥον
http://anond.hatelabo.jp/20120313004820
Slide 26
Slide 26 text
Mackerel
ʲՄࢉ໊ࢺʳɹαό
ʲෆՄࢉ໊ࢺʳαό(ͷ)
Plack / Starlet / Teng / Router::Simple /
Class::Accessor::Lite::Lazy / TheSchwartz / Text::Xslate /
Config::ENV / Scope::Container / SQL::Maker ...
Perl 5.14 / Carton 1.0
Slide 27
Slide 27 text
·ͣࢥ͔Β
Slide 28
Slide 28 text
ϗετใͷҰݩཧ
͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετใ͕͋ΔͷΑ͘ͳ͍
֤छπʔϧͱૄ݁߹ʹ౷߹
REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
Slide 29
Slide 29 text
ϗετใͷҰݩཧ
ϗετཧ
ཧϗετ or Ծϗετ or AWS
ཧϗετͱԾϗετͷରԠؔ
ͲͷαʔϏεͷͲ͏͍͏ׂ(DBαʔό,ϓϩΩγαʔόͳͲʣ
ͲͷσʔληϯλͷͲͷϥοΫ
ͲͷωοτϫʔΫʢαϒωοτʣ
ঢ়ଶ
(working,standby,maintenace,poweroff,destroyed)
Slide 30
Slide 30 text
ServiceҰཡ
Slide 31
Slide 31 text
ServiceҎԼͷRoleҰཡ
RoleҰཡ
Role
ϗετใ
Slide 32
Slide 32 text
ৄࡉϗετใ
load average CPU time
iops memory usaage
... other graphs
Slide 33
Slide 33 text
पลπʔϧͱͷ࿈ܞ
Slide 34
Slide 34 text
Nagios࿈ܞ
Slide 35
Slide 35 text
ઃఆͷࣗಈੜ
MackerelͷAPI͔ΒϗετใΛऔಘͯ͠ϗετఆٛ
ϑΝΠϧΛࣗಈੜ
NagiosͷϗετstatusΛMackerelͷϏϡʔʹදࣔ
NagiosͷWeb UIʹͱͳ͍͍ͯ͘Α͏ʹ
ৄ͘͠ίνϥ
ϞχΧδ#3 http://www.slideshare.net/shoichimasuhara/3-ss-17097401
Nagios࿈ܞ
Slide 36
Slide 36 text
Chef࿈ܞ
(Just Idea)
Slide 37
Slide 37 text
Chef Serverͱͷಉظ
MackerelͷDBΛϚελͱ͢Δ
Mackerelʹߋ৽ɾআ͕ΕChef ServerͷAPIΛୟ
͍ͯPOST
Chef Serverͷഇࢭ
MackerelͷRole͝ͱʹϨγϐΛ༻ҙ
Ϩγϐͷద༻ઌMackerelͷAPI͔Βऔಘ
Knife SoloͰϨγϐద༻
Slide 38
Slide 38 text
Capistrano࿈ܞ
Slide 39
Slide 39 text
Capistrano࿈ܞ
CapfileʹʹσϓϩΠઌϦετΛॻ͖ͨ͘ͳ͍
σϓϩΠ͢ΔͨͼʹMackerelͷAPIΛୟ͍ͯσϓϩΠઌ
ϗετҰཡΛऔಘ
http://mackerelhost/service/Hatena-Blog/roles/
backend?format=json
backend = ΞϓϦέʔγϣϯαʔό
ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷΑ͏ʹRoleͰࢦఆ
Slide 40
Slide 40 text
IRC࿈ܞ
Slide 41
Slide 41 text
IRC࿈ܞ
ϗετใͷ࡞ɾߋ৽ɾআΛIkachanܦ༝Ͱ
IRCʹ௨
୭͕ͲͷΑ͏ͳૢ࡞Λ͔͕ͨ͠ϦΞϧλΠϜʹΘ͔Δ
DBૢ࡞ʹରͯ͠௨ͷHookΛࠐΉ
(wtatsuru) update host 'intern2013': {"status"=>"maintenance"}
εςʔλεΛ”ϝϯςφεத”ʹҠߦ
Slide 42
Slide 42 text
DNS࿈ܞ
Slide 43
Slide 43 text
RoleΛߏ͢Δϗετ
reverse
proxy LVS
Role Role
Խɾෛՙࢄ͞Ε͍ͯΔ͜ͱʹண
Slide 44
Slide 44 text
Խɾෛՙࢄ
Role
ԾIP(VIP)
LVS
Role
DNS
ϥϯυϩϏϯ
DNS
DNSϥϯυϩϏϯ༻FQDNʹର
ͯ͠ෳͷIP͔ΒϥϯυϩϏ
ϯͰҰͭʹIPΛฦ͢
LVSϗετʹԾIPΛׂΓͯ
ԾIP͑͞Θ͔͍ͬͯΕɹɹ
͋ͱউखʹࢄ
Slide 45
Slide 45 text
ϥϯυϩϏϯ༻FQDNͱ
VIP͋·Γม͑ͨ͘ͳ͍
Service໊ͱRole໊΄ͱΜͲมΘΒ
ͳ͍͜ͱʹண
Slide 46
Slide 46 text
DNSϥϯυϩϏϯ
DNSϥϯυϩϏϯ༻ͷFQDNΛService-Roleϕʔε
ͷ໊લͰׂΓͯ
nslookup hosts.db-slave.bookmark
Name: hosts.db-slave.bookmark
Address: 10.0.0.1
Name: hosts.db-slave.bookmark
Address: 10.0.0.2
Name: hosts.db-slave.bookmark
Address: 10.0.0.3
role.service
Slide 47
Slide 47 text
VIP
LVSͷVIPʹରͯ͠FQDNΛService-Roleϕʔε
ͷ໊લͰׂΓͯ
nslookup db-slave.bookmark
Name: db-slave.bookmark
Address: 192.168.1.1
role.service
Slide 48
Slide 48 text
पลπʔϧͱͷ࿈ܞͷऴΘΓ
Slide 49
Slide 49 text
αʔόཧʹ͍ͨϏϡʔ
Slide 50
Slide 50 text
ใີ͕ߴ͘ཏతͳϏϡʔ
ڝഅ৽ฉΈ͍ͨͳΠϝʔδ
ۭ͖༰ྔͱ͔͕ͬ͟ͱΈΕΔ
Slide 51
Slide 51 text
ϗετ໊/IP
ͳͲ
loadavg
άϥϑ
Datacenter/
Rack
Service/
Role
Resource Memo
Slide 52
Slide 52 text
͍ͭΈΔάϥϑൺֱ͍ͨ͠άϥϑΛબΜͰ̍ϖʔδ
ʹ·ͱΊΔ
HTMLʹՃ͑ͯɺάϥϑه๏ΛͬͯΧελϚΠζ
[:graph:service=SERVICE:role=ROLE]
[:graph:service=SERVICE:role=ROLE]
ΧελϚΠζͰ͖ΔμογϡϘʔυ
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
Roleϗετͷάϥϑൺֱ
ͯͳϒϩάͷΞϓϦέʔγϣϯαʔόͷload averageൺֱ
ҟৗͳϗετ͕͍ͳ͍͔Ͳ͏͔ݕग़
Slide 55
Slide 55 text
લ͓ΘΓ
Slide 56
Slide 56 text
αʔόϝτϦΫεՄࢹԽͷ
Έ
ϝτϦΫεMPBEBWFSBHFɼ$16UJNFɼNFNPSZVTBHF
FUD
Slide 57
Slide 57 text
αʔόϝτϦΫεՄࢹԽͬͯʁ
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
MackerelͰͲ͏࣮ͬͯɹ
ͯ͠Δ͔ͷ
Slide 60
Slide 60 text
αʔόϝτϦΫεՄࢹԽʹඞਢͷཁૉ
Slide 61
Slide 61 text
αʔόϝτϦΫεՄࢹԽ
Host
Host
Host
Host
Process
Storage
֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
ετϨʔδʹ֨ೲ
ετϨʔδ͔ΒΛऔΓग़͢
άϥϑԽ
Slide 62
Slide 62 text
ͲͷΑ͏ͳΞʔΩςΫνϟ͕Α͍͔
Slide 63
Slide 63 text
Push or Pull
Slide 64
Slide 64 text
Pushܕ
Host
Host
Host
Host
Process
Storage
֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
Host͔ΒProcess
ϝτϦΫεΛPush
Slide 65
Slide 65 text
Pushܕͷ
֤ϗετʹcronͰճ͢ϝτϦΫεऔಘ&ૹ৴εΫ
ϦϓτΛ͠ͳ͚ΕͳΒͳ͍
εΫϦϓτͷܧଓతͳมߋ͕໘
Slide 66
Slide 66 text
Pullܕ
Host
Host
Host
Host
Process
Storage
֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
Process͔ΒHost
ϦΫΤετ͢Δ
Slide 67
Slide 67 text
PullܕͰ݁ہࣗલͷ
ʢܧଓతͳมߋ͕ඞཁͳʣσʔϞϯΛ
ཱͯͳ͚ΕͳΒͳ͍ͷͰʁ
Slide 68
Slide 68 text
ϝτϦΫεऔಘͷͨΊͷ
ωοτϫʔΫΠϯλϑΣʔεΛͭ
طଘͷσʔϞϯΛ͏
Slide 69
Slide 69 text
SNMP
OS͕ఏڙ͢ΔجຊతͳϝτϦΫεΛऔಘ
ͦͷଞϧʔλSquidͳͲ
Net::SNMPΛ͏
my ($session, $error) = Net::SNMP->session(
-hostname => $hostname,
-community => $community,
-version => 2,
-timeout => 10,
-translate => 0x0,
);
$session // croak "SNMP error: $error";
my $response = $session->get_request(
-varbindlist => $mibs, # MIBΛෳࢦఆ
) || croak "SNMP error: $session->error";
Slide 70
Slide 70 text
Nginx
HttpStubStatusModule
HTTPͰNginxݻ༗ͷϝτϦΫεΛͱΕΔ
Active connections: 291
server accepts handled requests
16630948 16630948 31070465
Reading: 6 Writing: 179 Waiting: 106
$ curl http://nginxhost/nginx_status:8080
Slide 71
Slide 71 text
Plack
Plack::Middleware::ServerStatus::Lite
JSON format͕͋ΔͷͰָ
% curl http://server:port/server-status?json
{"Uptime":"1332476669","BusyWorkers":"2",
"stats":[
{"protocol":null,"remote_addr":null,"pid":"78639",
"status":".","method":null,"uri":null,"host":null,"ss":null},
{"protocol":"HTTP/
1.1","remote_addr":"127.0.0.1","pid":"78640",
"status":"A","method":"GET","uri":"/","host":"localhost:
10226","ss":0},
...
],"IdleWorkers":"3"}
Slide 72
Slide 72 text
Memcached
TelnetΠϯλϑΣʔε
stats
STAT pid 14868
STAT uptime 175931
STAT time 1220540125
STAT version 1.2.2
...
STAT curr_connections 92
STAT total_connections 1740
STAT connection_structures 165
STAT cmd_get 7411
STAT cmd_set 28445156
STAT get_hits 5183
STAT get_misses 2228
STAT evictions 0
STAT bytes_read 2112768087
STAT bytes_written 1000038245
STAT limit_maxbytes 52428800
STAT threads 1
END
my $sock = IO::Socket::INET->new(
PeerAddr => "$hostname:$port",
Proto => 'tcp',
Timeout => 10,
) or croak "Couldn't connect to $hostname:$port";
$sock->print("stats\r\n");
my $value_by_stat = {};
while (my $line = $sock->getline) {
last if $line =~ /^END/;
$line =~ s/\n$|\r\n$//; #chomp
if ($line =~ /^STAT\s+(\S*)\s+(.*)/) {
$value_by_stat->{$1} = $2;
}
}
Slide 73
Slide 73 text
Pullܕ (࠶ܝ)
Host
Host
Host
Host
Process
Storage
֤ϗετ͔ΒͷϝτϦΫεͷऔಘ
ετϨʔδʹ֨ೲ
ετϨʔδ͔ΒΛऔΓग़͢
Slide 74
Slide 74 text
Process
Slide 75
Slide 75 text
Process
ϗετͨ͘͞Μ͋ΔͷͰฒྻ(ඇಉظ)Ͱɹɹɹ
ϝτϦΫεऔಘ͍ͨ͠
͔͠͠ɺϗετͷϓϩηεΛཱͯΔͷ
ͪΐͬͱ...
PerlͰεϨου...
Slide 76
Slide 76 text
WorkerManager
Job Queue(The Schwartz)ʹରԠͨ͠ϓϩηεཧ
σʔϞϯ
Preforkܕ: ϓϩηεΛ੍ݶͰ͖Δ
͔ΘΓʹParallel::PreforkStarlet/Starman
ΛͬͯΑ͍
Slide 77
Slide 77 text
Jobͷॲཧ༰
Slide 78
Slide 78 text
Agent
֤JobΛAgentͱݺΜͰ͍Δ
ϦϞʔτϗετ͔ΒϝτϦΫεΛͱͬͯ͘Δϓϩηε
ରԠϛυϧΣΞɼϓϩτίϧ
Apache, Nginx, MySQL, Munin, Latency,
Plack, Perlbal, Redis, SNMP, Solr,
TheSchwartz
ϓϥΨϒϧ
Hatena::Mackerel::Worker::Agent::XXX
SNMPHTTP, TelnetͰϝτϦΫεऔಘ
Slide 79
Slide 79 text
ϓϥάΠϯPerlͰͳ͍ͱॻ͚ͳ͍
ݴޠඇґଘͰϓϥάΠϯΛॻ͖͍ͨ
Slide 80
Slide 80 text
ͦΕMunin͡ΌͶ
MuninͷϓϥάΠϯ
ඪ४ೖग़ྗͰύϥϝʔλͷड͚͠Λ͢Δ
Slide 81
Slide 81 text
munin-nodeͱͷ࿈ܞ
munin-nodeΛϦϞʔτϗετʹཱ͓ͯͯ͘
mackerel agent͕TelnetͰmunin pluginͷ݁Ռऔಘ
munin-node
munin plugin
mackerel
agent
telnet
ඪ४ೖग़ྗ
Slide 82
Slide 82 text
munin-nodeͱซ༻
ϛυϧΣΞ͕ϝτϦΫεऔಘͷͨΊͷωοτϫʔ
ΫΠϯλϑΣʔε͕͋ΔͳΒ
mackerel agent͔ΒωοτϫʔΫɾΠϯλ
ϑΣʔεΛ͏
netstatsmartctlͳͲϦϞʔτϗετͷίϚϯ
υΛୟ͖͍ͨͳΒ
munin-nodeΛ͏
Slide 83
Slide 83 text
ϝτϦΫεͷߋ৽
(ετϨʔδͷ)
Slide 84
Slide 84 text
Pullܕ (࠶ܝ)
Host
Host
Host
Host
Worker
Manager
Storage
Slide 85
Slide 85 text
Storage
RDBMS, KVS, other NOSQL...
Slide 86
Slide 86 text
RRDtool
Slide 87
Slide 87 text
RRD (Round Robin Database)
࣌ܥྻσʔλΛ֨ೲ͢ΔͨΊͷσʔλϑΥʔϚοτ
อଘظؒΛݻఆͯ͠ݹ͍ࣺ͍ͯͯ͘
RRDtool
RRDͷ࣮ʢσϑΝΫτελϯμʔυʣ
RRDͷ࡞ɾߋ৽ɾࢀরػೳͱάϥϑԽػೳΛͭ
I/O͕ͨΒ͍(Β͍͠)
େͷαʔόཧπʔϧͰΘΕ͍ͯΔ
ͳ͔ͥάϥϑը૾ੜػೳ͕͋Δ
Slide 88
Slide 88 text
RRDtoolϝϦοτɾσϝϦοτ
ϝϦοτ
࣌ܥྻσʔλΛ͍͍͔Μ͡ʹॻ͖ࠐΜͰ͘ΕΔ
ߋ৽ִ͔ؒΒ࣮ࡍͷॻ͖ࠐΈ࣌ࠁ͕গʑͣΕͯमਖ਼ͯ͘͠
ΕΔ (10:01 -> 10:00)
ͷਫ਼ͷௐઅ͕Մೳ
ϥϯυϩϏϯͳͷͰϑΝΠϧαΠζ͕Ұఆ
I/O͕ߴ
σϝϦοτ
rrdtoolʹ༩͑Δύϥϝʔλͷઃఆ͕ࡶ
Slide 89
Slide 89 text
ొਓ͕ͰͦΖͬͨͷͰ
γεςϜશମΛཧ
Slide 90
Slide 90 text
Job Queue
DB
RRDtool
Host
Host
Host
Host
enqueue.pl
cron
δϣϒೖ
ϗετใऔಘ
ϝτϦΫεऔಘ
Slide 91
Slide 91 text
RRDtool
App Server
Browser
HTTP graph
request
rrdtool graph
άϥϑϦΫΤετ
Slide 92
Slide 92 text
RRDtool
App Server
Browser
HTTP graph
response
graph image
άϥϑϨεϙϯε
Slide 93
Slide 93 text
Job Queue
DB
RRD
Host
Host
Host
Host
enqueue.pl
cron
RRDtool
App Server
Browser
γεςϜશମ
Slide 94
Slide 94 text
Web͔Βάϥϑੜͤ͞Δ
Έ
Slide 95
Slide 95 text
RRD
App Server
Browser
HTTP graph
request
rrdtool graph
άϥϑϦΫΤετ
ʢ࠶ܝʣ
Slide 96
Slide 96 text
RRD
App Server
Browser
HTTP graph
request
rrdtool graph
άϥϑϦΫΤετ
ʢ࠶ܝʣ
άϥϑੜURLΛͲ͏දݱ
͢Δ͔
Slide 97
Slide 97 text
ཁrrdtool graphίϚϯυͷҾ
ΛURLͰͲ͏දݱ͢Δ͔
Slide 98
Slide 98 text
rrdtool graph
$ rrdtool graph --end now --start end-120000s --width 400 \\
DEF:ds0a=/home/rrdtool/data/router1.rrd:ds0:AVERAGE \\
DEF:ds0b=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=1800 \\
DEF:ds0c=/home/rrdtool/data/router1.rrd:ds0:AVERAGE:step=7200 \\
LINE1:ds0a#0000FF:"default resolution\l" \\
LINE1:ds0b#00CCFF:"resolution 1800 seconds per interval\l" \\
LINE1:ds0c#FF00FF:"resolution 7200 seconds per interval\l"
άϥϑඳըظؒͷࢦఆ
άϥϑ෯ࢦఆ
σʔλϦιʔεʢRRDϑΝΠϧ໊ʣࢦఆ
άϥϑઢͷଠ͞৭ɺຌྫΛࢦఆ
...
Slide 99
Slide 99 text
γϯϓϧURL
ͱ
RRDtool URL
ͷ2छྨ༻ҙ
Slide 100
Slide 100 text
γϯϓϧURL
/graph?
define=tesuhost01::_default::loadavg5::STACK,testhost0
2::mysql::second_behind_master
define=::::,
define=::::,
...
σϑΥϧτΛܾΊͯɺͳΔ͘γϯϓϧʹ
ઢͷଠ͞ͱ͔σϑΥϧτΛ෦ͰܾΊ͓ͯ͘
Slide 101
Slide 101 text
RRDtool URL
RRDtoolίϚϯυΛͦͷ··URDͰදݱ
hit rateͷܭࢉͱ͔Λͤ͞Δͱ͖ʹ͏
/graph/general?s=[
(def:cmd_get:::=path:memcachedhost,memcached,cmd_get:value:AVERAGE),
(def:get_hits:::=path:memcachedhost,memcached,get_hits:value:AVERAGE),
(cdef:hit_rate:::=get_hits,cmd_get,/),
(line1:hit_rate:::@0000ff:memcachedhost-memcached-hit_rate)
!end=now-0d,height=100,start=now-30d,width=400
]
memcachedͷhit rateάϥϑURL
Slide 102
Slide 102 text
my $fmt = <<'EOS';
graphformula = "[graph:" ( graphs [ ":" option ] ) "]"
graphs = graph | "(" graph ")" 0*( ",(" graph ")" )
graph = "path:" elements | instruction ":" [ name ] ":::" option )
elements = object : [ tag : [ label ] ]
object = name
tag = name
label = name
instruction = "def" | "cdef" | "vdef" | "line" digit | "area" | "hrule" | "vrule" | "print" |
"gprint" | "commend" | "tick" | "shift" | "textalign"
option = 1*( char | "," | "=" | "#" | "@" | ":" | " " | "\" | op)
name = 1*char
char = alphanum | mark
op = "+" | "/" | "%" | "*"
mark = "-" | "_" | "." | "!" | "~" | "*" | "'"
alphanum = alpha | digit
alpha = lowalpha | upalpha
lowalpha = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" |
"j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" |
"s" | "t" | "u" | "v" | "w" | "x" | "y" | "z"
upalpha = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" |
"J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" |
"S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"
digit = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" |
"8" | "9"
EOS
BNFͰਅ໘ʹ༷Λఆٛ
Slide 103
Slide 103 text
ύϑΥʔϚϯεվળ
Slide 104
Slide 104 text
RRDͷσΟεΫI/Oෛՙ
(ϗετ) x (ϝτϦΫεछผ) io per 5
ϗετ = ઍ
ϝτϦΫεछผ = 20Ҏ্
10k io per 5
࣮ࡍ5ͷ͏ͪͷ͋Δ࣌ؒʹI/O͕ूத
Slide 105
Slide 105 text
2छྨͷෛՙݮํ๏
Slide 106
Slide 106 text
rrdcached
rrdtoolͷߋ৽ूॲཧΛड͚͚ΔσʔϞϯ
rrdͷߋ৽ಉ࣌͡ࠁʹࡴ౸͕ͪ͠
ॻ͖ࠐΈΛϥϯμϜͳ࣌ؒԆͯ͘͠ΕΔ
rrdcachedͳͩ͠ͱ, ฏۉ 700 iops όʔετ
1200 iops
SSDͩͱͳΜͱ͔ͳΔ HDD(AWS)ͩͱݫ͍͠
ϦϦʔε൛(1.4.8)updateʹ͔͠ରԠ͍ͯ͠ͳ͍
͕trunkͳΒͦͷଞͷ֤छαϒίϚϯυʹରԠ
Slide 107
Slide 107 text
ࢄRRD
RRDͷσΟεΫI/OΛϦϞʔτϗετ୯ҐͰࢄ
͋Δϗετ͕ͲͷRRDϗετʹׂΓͯΒΕ͍ͯΔ
͔ͷใΛDBʹ͓࣋ͬͯ͘
Job Queue
DB
RRD
enqueue.pl
Job Queue RRD
Job Queue RRD
Slide 108
Slide 108 text
Future Work
Slide 109
Slide 109 text
mackereld
Slide 110
Slide 110 text
mackereld
ϝτϦΫεऔಘͷͨΊͷσʔϞϯ
munin-node૬ͷػೳ
REST APIͰϝτϦΫεऔಘ
ϦϞʔτϗετͷϩάΛWebͰϦΞϧλΠϜɹɹ
ʹΈΕΔ
֤छϩάϑΝΠϧΛࢹ͠ɺมߋΛݕͨ͠Β
Server-Sent EventsͰͱ͢
ܧଓతͳมߋ͕ΓΊΜͲ͏
Slide 111
Slide 111 text
RRDtoolʹΑΔ༧ଌάϥϑ
Slide 112
Slide 112 text
RRDtoolʹΑΔ༧ଌάϥϑ
σΟεΫ͕ຬഋʹͳΔͷ͍ͭʁ
20144݄
Slide 113
Slide 113 text
$ rrdtool graph ./dskUsed.png \
--start="now-1y" \
--end="now+2y" \
--imgformat=PNG \
--title="Disk Usage" \
--height=200 \
--width=500 \
--lower-limit=0 \
DEF:usage=testhost__dskUsed.rrd:value:MAX \
DEF:total=testhost__dskTotal.rrd:value:MAX \
CDEF:c_warn=total,0.85,* \
CDEF:c_crit=total,0.95,* \
VDEF:v_total=total,MINIMUM \
VDEF:v_warn=c_warn,MINIMUM \
VDEF:v_crit=c_crit,MINIMUM \
VDEF:v_usage_slope=usage,LSLSLOPE \
VDEF:v_usage_intercept=usage,LSLINT \
CDEF:c_usage_predict=usage,POP,v_usage_slope,COUNT,*,v_usage_intercept,+ \
CDEF:c_rwarn=c_usage_predict,v_warn,v_total,LIMIT \
VDEF:v_rwarn=c_rwarn,FIRST \
CDEF:c_rcrit=c_usage_predict,v_crit,v_total,LIMIT \
VDEF:v_rcrit=c_rcrit,FIRST \
HRULE:v_warn#FF8800:"warning":dashes=5 \
VRULE:v_rwarn#FF8800::dashes=5 \
HRULE:v_crit#FF4400:"critical" \
VRULE:v_rcrit#FF4400 \
HRULE:v_total#FF0000:"total" \
AREA:usage#00FF00:"Disk Usage" \
LINE1:c_usage_predict#0000FF:"Predict" \
GPRINT:v_rwarn:"Reach warning (85%)\: %c":strftime \
GPRINT:v_rcrit:"Reach critical (95%)\: %c":strftime
shoichimasuhara++
Slide 114
Slide 114 text
Rich Visualization
Slide 115
Slide 115 text
Rich Visualization
তͬΆ͍(ը૾)
ฏੜ·Εͬͪ͜(SVG)
RRDtool
D3.js
Slide 116
Slide 116 text
D3.js
NVD3.js (Re-usable charts for d3.js)
Rickshaw (time-series graph)
rrdtool xport --json ͰJSONܗࣜͰΛऔಘɹɹ
Ͱ͖ΔͨΊɺετϨʔδRRDͷ··
D3.js is a JavaScript library for manipulating documents based on data.
D3 helps you bring data to life using HTML, SVG and CSS.
Slide 117
Slide 117 text
RRDtool͔Βͷ٫
Slide 118
Slide 118 text
ΘΓʹRedisMongoDBͳͲͷετϨʔδΛ͏
FnordMetrics (http://fnordmetric.io/)
GrowthForecastʹࣅͯΔ
ετϨʔδɿRedis
άϥϑɿRickshaw
my $furl = Furl->new(timeout => 10);
my $res = $furl->post(
"http://fnordmetricshost:4242/events",
['Content-Type' => 'application/json'],
encode_json({
"_type" => '_incr',
"value" => scalar @$statuses,
"gauge" => "tweets_per_minute"
}),
);
Slide 119
Slide 119 text
αʔόϝτϦΫεՄࢹԽͷ
͓ΘΓ
Slide 120
Slide 120 text
·ͱΊ
Slide 121
Slide 121 text
ϗετใͷҰݩཧ
͋ͬͪͬͪ͜ʹಉ͡Α͏ͳϗετใ͕͋ΔͷΑ͘ͳ͍
֤छπʔϧͱૄ݁߹ʹ౷߹
REST APIʹΑΔपลπʔϧͱͷ࿈ܞ
Slide 122
Slide 122 text
ϝτϦΫεՄࢹԽͷ࣮
PullܕΞʔΩςΫνϟ
WorkerManagerͰϓϩηεཧ
StorageʹRRDtoolΛ͏
RRDtoolΜΓ
RRDtool͍ͨΜ
RRDtool͍
RRDtoolΉ͔͍ͣ͠
Slide 123
Slide 123 text
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
Thanks to id:stanaka, id:halfrack, id:wtatsuru, id:shoichimasuhara, id:hagihala