Slide 1

Slide 1 text

3 popular Ops Tools in Japan SAKAMOTO Takumi @takus Ignite Velocity 2013

Slide 2

Slide 2 text

Background •  There are a lot of large scale web services in Japan ü  > tens of millions users, > thousands of servers e.g. mobage by DeNA •  social game platform •  3.5 billions PV/day •  thousands of servers

Slide 3

Slide 3 text

Server Provisioning System Monitoring Performance Tuning Log Management Capacity Plan Japanese engineers have developed tools for ...

Slide 4

Slide 4 text

Test-Driven Infrastructure If services are provisioned by programs, then we can - and should - test those programs. Further, we can write them in a test-driven style: start with the description of how the service should behave, and write a test that expresses this. Then write code that makes that test pass. '' " http://bitfieldconsulting.com/review-test-driven-infrastructure-chef by John Arundel

Slide 5

Slide 5 text

serverspec RSpec Tests for Provisioned Servers 1 cat spec/www.example.jp/httpd_spec.rb 2 describe package('httpd') do 3 it { should be_installed } 4 end 5 describe service('httpd') do 6 it { should be_running } 7 end 8 9 describe file('/etc/httpd/conf/httpd.conf') do 10 it { should be_file } 11 it { should contain 'ServerName www.example.jp' } 12 end 13 14 rake spec 15 /usr/bin/ruby -S rspec spec/www.example.jp/httpd_spec.rb 16 ...... 17 Finished in 0.99715 seconds 18 6 examples, 0 failures  

Slide 6

Slide 6 text

A Lot of Built in Matchers should be_file should be_direcotry should be_linked_to ... file should be_resolvable should be_resolbavle.by(host) should be_resolbavle.by(dns) host should have_rule(-P INPUT ACCEPT) should be_resolbavle.by(host) should be_resolbavle.by(dns) iptables And More ... should be_installed should be_installed.by(gem) should be_installed.with_version(1.0) package should be_enabled should be_running should be_running.under('supervisor') service should exist should be_belong_to_group (wheel) should be_uid(1105) user

Slide 7

Slide 7 text

Use Case •  Migrating to/Introducing new provisioning framework ü  write serverspec for defining how the server should be •  independent test of provisioning framework ü  write cookbook/manifest/shell script •  serverspec assures that new PF builds the server correctly

Slide 8

Slide 8 text

Why serverspec? •  Light-Weight ü  start to use without any provisioning framework ü  easy to use & understand the behavior ü  no required agent (using ssh & shell command) •  Independent from Any Provisioning Frameworks ü  flexible, avoiding lock in ...

Slide 9

Slide 9 text

Visualizing Metrics Imagine designing a car without any of the dials or indicators in front of the driver. Now paint the windshield. That's what it's like to run a web operation without metrics. '' " Web Operations: Keeping the Data On Time by John Allspaw

Slide 10

Slide 10 text

GrowthForecast Application Metrics Visualizer 1 $ crontab -l 2 */5 * * * * curl -F number=`mysql -BN -e 'select count(*) from member' game` 3 http://example.com/api/socialgame/member/register 2>&1 >/tmp/post.log You can visualize data in MySQL with the following one-liner

Slide 11

Slide 11 text

•  Visualizing everything via Web API ü  include layered graphs Use Case POST /api/service1/web/2xx_count POST /api/service1/web/3xx_count POST /api/service1/web/4xx_count POST /api/service1/web/5xx_count Growth Forecast all_http_status_count Create Layered Graph 3xx_count 2xx_count 4xx_count 5xx_count

Slide 12

Slide 12 text

Why GrowthForecast? •  Easy to install ü  2 yum command & 1 cpanm command (on RHEL6) •  Update metrics & get graphs by HTTP ü  easy to integrate with any systems •  Regular data size (∵ RRDTool) ü  free from disk usage planning

Slide 13

Slide 13 text

Log Management Consider a user beyond a single line of access log. We are not sure whether a user becomes happy by a 200 response code line. However, we are sure that a user becomes unhappy by a 500 response code line. '' " http://ihara2525.tumblr.com/post/17029509298 by Masahiro Ihara (translated by me)

Slide 14

Slide 14 text

Fluentd Pluggable Log Management Tool •  Enables you to build your own log solutions easily ü  structured logging (treat log as JSON) ü  pluggable architecture input buffer output File Tail HTTP dstat File Memory File Amazon S3 HDFS GrowthForecast

Slide 15

Slide 15 text

Use Case Server Server Server Worker Amazon S3 hourly Archive GrowthForecast Visualizing Server Server Server MongoDB Searching recent logs forward access logs

Slide 16

Slide 16 text

type forward name worker host 192.168.1.3 port 24224 forward to worker Server (input logs) type tail path /var/log/httpd-access.log pos_file /var/log/access.log.pos tag app1.access format apache2 input from log file type tail path /var/log/httpd-access.log pos_file /var/log/access.log.pos tag app2.access format apache2

Slide 17

Slide 17 text

type s3 s3_bucket apache_access s3_endpoint us-west-1 path logs/ type mongo host fluentd port 27017 database apache collection access type datacounter count_interval 60 aggregate tag output_per_tag yes tag_prefix datacounter count_key status pattern1 2xx ^2\d\d$ pattern2 3xx ^3\d\d$ pattern4 4xx ^4\d\d$ pattern5 5xx ^5\d\d$ type growthforecast gfapi_url http://gf.com/api/ service service1:web tag_for section name_keys 2xx_count,3xx_count, 4xx_count,5xx_count Worker (output logs) fluent-plugin-datacounter fluent-plugin-growthforecast type forward port 24224 bind 0.0.0.0 fluent-plugin-mongo fluent-plugin-S3

Slide 18

Slide 18 text

Why Fluentd? •  Real Time Log Processing ü  you can know what’s going on. batch operation is too late. •  Extensibility ü  input, buffer, output are pluggable, a lot of plugins exist •  Reliable ü  HA configuration (re-send, failover)

Slide 19

Slide 19 text

serverspec RSpec Tests for Provisioned Servers GrowthForecast Application Metrics Visualizer Fluentd Pluggable Log Management Tool 3 popular Ops Tools in Japan

Slide 20

Slide 20 text

Thank you