Upgrade to Pro — share decks privately, control downloads, hide ads and more …

agri inventory - nouka data collector / yaoya data convertor

agri inventory - nouka data collector / yaoya data convertor

overview of agri inventory. composed by nouka data collector, yaoya data convertor

(スライドのホストをSlideShareから引っ越し)
https://www.slideshare.net/toshiak_netmark/agri-inventory-nouka-data-collector-yaoya-data-convertor

Toshiaki Baba

June 17, 2012
Tweet

More Decks by Toshiaki Baba

Other Decks in Programming

Transcript

  1. agri inventory nouka data collector naya data store yaoya data

    convertor nouka naya yaoya output push crawl
  2. yaoya output example(1) ¤ $ bin/yaoya --group system_A --name command_uname

    {u'return_code': u'0', u'command_line': u'uname -a', u'execute_at': u'1339942622', u'group_name': u'system_A', u'visible': u'True', u'host_name': u'bb-mba.local', u'time': datetime.datetime(2012, 6, 17, 14, 17, 3), u'output': u'Darwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386¥n', u'_id': ObjectId('4fdde6e0c286630ca0000009'), u'command_name': u'command_uname'} {u'return_code': u'0', u'command_line': u'uname -a', u'execute_at': u'1339943547', u'group_name': u'system_A', u'visible': u'True', u'host_name': u'bbtest-c6', u'time': datetime.datetime(2012, 6, 17, 14, 32, 28), u'output': u'Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/Linux¥n', u'_id': ObjectId('4fddea7dc286630ca000003c'), u'command_name': u'command_uname'}
  3. yaoya output example(2) ¤ $ bin/csv --group system_A --name command_uname

    "bb-mba.local","Darwin bb-mba.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu- 1504.15.3~1/RELEASE_I386 i386" "bbtest-c6","Linux bbtest-c6 2.6.32-220.13.1.el6.x86_64 #1 SMP Tue Apr 17 23:56:34 BST 2012 x86_64 x86_64 x86_64 GNU/Linux"
  4. default commands ¤ command_ifconfig = ifconfig -a ¤ command_hostname =

    hostname ¤ command_uname = uname -a ¤ command_df = df -akl ¤ command_iptables = /sbin/iptables-save ¤ command_rpm = rpm -qa ¤ command_chkconfig = /sbin/chkconfig --list ¤ command_ip_addr = /sbin/ip addr show ¤ command_ip_route = /sbin/ip route show ¤ command_dmidecode = /usr/sbin/dmidecode ¤ command_resolv = cat /etc/resolv.conf ¤ command_passwd = cat /etc/passwd ¤ command_group = cat /etc/group ¤ command_shadow = cat /etc/shadow ¤ command_proc_cpuinfo = cat /proc/cpuinfo ¤ command_proc_meminfo = cat /proc/meminfo ¤ command_proc_mounts = cat /proc/mounts To add command, just only define in nouka.conf!
  5. nouka data collector ¤ pure python 2.4 ¤ run with

    python standard library ¤ POST result to naya with HTTP
  6. naya data store ¤ fluentd / td-agent ¤ fluent-plugin-http-enhanced ¤

    https://github.com/parolkar/fluent-plugin-http-enhanced ¤ fluent-plugin-mongo ¤ mongodb
  7. data format { *** are same value in same execution

    ‘group_name’ : ‘<group_name’s value in config file>‘, *** 'host_name' : '<env HOSTNAME>', *** 'command_name' : '<command name in config file. named command_*>', 'command_line' : '<command line(value) in config file>', 'output' : '<output of command line>', 'return_code' : '<return code of command line>', ‘visible’ : ‘True’, *** 'execute_at' : '<execute date as unixtime>’, *** }
  8. setup(1/2) ¤ td-agent ¤ cat <<EOT >/etc/yum.repos.d/td.repo [treasuredata] name=TreasureData baseurl=http://packages.treasure-

    data.com/redhat/¥$basearch gpgcheck=0 EOT ¤ yum -y install td-agent ¤ /usr/lib64/fluent/ruby/bin/gem install bson_ext ¤ /usr/lib64/fluent/ruby/bin/gem install fluent-plugin-http- enhanced
  9. setup(2/2) ¤ mongod ¤ cat <<EOT >/etc/yum.repos.d/10gen.repo [10gen] name=10gen Repository

    baseurl=http://downloads- distro.mongodb.org/repo/redhat/os/x86_64 gpgcheck=0 EOT ¤ yum -y install mongo-10gen.x86_64 mongo-10gen-server.x86_64 ¤ mercurial ¤ yum -y install python-devel python-setuptools gcc ¤ pip install mercurial