agri inventory - nouka data collector / yaoya data convertor

overview of agri inventory. composed by nouka data collector, yaoya data convertor

（スライドのホストをSlideShareから引っ越し）
https://www.slideshare.net/toshiak_netmark/agri-inventory-nouka-data-collector-yaoya-data-convertor

Transcript

1. agri inventory
   

   View Slide

2. agri inventory
   nouka data collector
   naya data store
   yaoya data convertor
   nouka
   naya
   yaoya
   output
   push crawl
   

   View Slide

3. yaoya output example(1)
   ¤ $ bin/yaoya --group system_A --name command_uname
   {u'return_code': u'0', u'command_line': u'uname -a', u'execute_at':
   u'1339942622', u'group_name': u'system_A', u'visible': u'True', u'host_name':
   u'bb-mba.local', u'time': datetime.datetime(2012, 6, 17, 14, 17, 3), u'output':
   u'Darwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7
   16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386¥n', u'_id':
   ObjectId('4fdde6e0c286630ca0000009'), u'command_name':
   u'command_uname'}
   {u'return_code': u'0', u'command_line': u'uname -a', u'execute_at':
   u'1339943547', u'group_name': u'system_A', u'visible': u'True', u'host_name':
   u'bbtest-c6', u'time': datetime.datetime(2012, 6, 17, 14, 32, 28), u'output':
   u'Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST
   2012 x86_64 x86_64 x86_64 GNU/Linux¥n', u'_id':
   ObjectId('4fddea7dc286630ca000003c'), u'command_name':
   u'command_uname'}
   

   View Slide

4. yaoya output example(2)
   ¤ $ bin/csv --group system_A --name command_uname
   "bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin Kernel
   Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-
   1504.15.3~1/RELEASE_I386 i386"
   "bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1
   SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64
   GNU/Linux"
   

   View Slide

5. default commands
   ¤ command_ifconfig = ifconfig -a
   ¤ command_hostname = hostname
   ¤ command_uname = uname -a
   ¤ command_df = df -akl
   ¤ command_iptables = /sbin/iptables-save
   ¤ command_rpm = rpm -qa
   ¤ command_chkconfig = /sbin/chkconfig --list
   ¤ command_ip_addr = /sbin/ip addr show
   ¤ command_ip_route = /sbin/ip route show
   ¤ command_dmidecode = /usr/sbin/dmidecode
   ¤ command_resolv = cat /etc/resolv.conf
   ¤ command_passwd = cat /etc/passwd
   ¤ command_group = cat /etc/group
   ¤ command_shadow = cat /etc/shadow
   ¤ command_proc_cpuinfo = cat /proc/cpuinfo
   ¤ command_proc_meminfo = cat /proc/meminfo
   ¤ command_proc_mounts = cat /proc/mounts
   To add command, just only
   define in nouka.conf!
   

   View Slide

6. components
   

   View Slide

7. nouka data collector
   ¤ pure python 2.4
   ¤ run with python standard library
   ¤ POST result to naya with HTTP
   

   View Slide

8. naya data store
   ¤ fluentd / td-agent
   ¤ fluent-plugin-http-enhanced
   ¤ https://github.com/parolkar/fluent-plugin-http-enhanced
   ¤ fluent-plugin-mongo
   ¤ mongodb
   

   View Slide

9. data format
   { *** are same value in same execution
   ‘group_name’ : ‘‘, ***
   'host_name' : '', ***
   'command_name' : '',
   'command_line' : '',
   'output' : '',
   'return_code' : '',
   ‘visible’ : ‘True’, ***
   'execute_at' : '’, ***
   }
   

   View Slide

10. getting started
    

    View Slide

11. project repository
    ¤ nouka
    ¤ https://bitbucket.org/netmarkjp/nouka
    ¤ yaoya
    ¤ https://bitbucket.org/netmarkjp/yaoya
    

    View Slide

12. setup(1/2)
    ¤ td-agent
    ¤ cat </etc/yum.repos.d/td.repo
    [treasuredata]
    name=TreasureData
    baseurl=http://packages.treasure-
    data.com/redhat/¥$basearch
    gpgcheck=0
    EOT
    ¤ yum -y install td-agent
    ¤ /usr/lib64/fluent/ruby/bin/gem install bson_ext
    ¤ /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http-
    enhanced
    

    View Slide

13. setup(2/2)
    ¤ mongod
    ¤ cat </etc/yum.repos.d/10gen.repo
    [10gen]
    name=10gen Repository
    baseurl=http://downloads-
    distro.mongodb.org/repo/redhat/os/x86_64
    gpgcheck=0
    EOT
    ¤ yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64
    ¤ mercurial
    ¤ yum -y install python-devel python-setuptools gcc
    ¤ pip install mercurial
    

    View Slide

14. enjoy development!
    

    View Slide