Save 37% off PRO during our Black Friday Sale! »

glusterfs-pmux-en

254c7a2e2055fe9300ea3b419c4b1cfe?s=47 maebashi
November 13, 2013

 glusterfs-pmux-en

254c7a2e2055fe9300ea3b419c4b1cfe?s=128

maebashi

November 13, 2013
Tweet

Transcript

  1. ©ɹ2013 Internet Initiative Japan Inc. ©ɹ2013 Internet Initiative Japan Inc.

    Pmux: a lightweight MapReduce framework Using GlusterFS maebashi@iij.ad.jp https://speakerdeck.com/maebashi/glusterfs-pmux-en
  2. ©ɹ2013 Internet Initiative Japan Inc. About me •  Takahiro Maebashi

    •  Internet Initiative Japan Inc. (IIJ) •  ITpro: ITݕূϥϘ -- ෼ࢄϑΝΠϧγεςϜͷ GlusterFSɿ͜Μͳͱ͖ɺͲ͏ͳΔ – http://itpro.nikkeibp.co.jp/ article/COLUMN/20130104/447701/!
  3. ©ɹ2013 Internet Initiative Japan Inc. GlusterFS @ IIJ Tokyo Osaka

    Matsue
  4. ©ɹ2013 Internet Initiative Japan Inc. Proprietary container unit, IZmo (Matsue

    Data Center Park) IT module air-conditioning unit
  5. ©ɹ2013 Internet Initiative Japan Inc. GlusterFS servers in IZmo • 

    The racks within the module are set at an angle
  6. ©ɹ2013 Internet Initiative Japan Inc. Today's Talk

  7. ©ɹ2013 Internet Initiative Japan Inc. glusterfs-hadoop

  8. ©ɹ2013 Internet Initiative Japan Inc. What is MapReduce? Represent problems

    as Map and Reduce step (1) Map – extract, convert (2) Reduce – aggregate, summarize
  9. ©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS?

  10. ©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS? (2) (in

    case of distributed volume) locate files based solely on their name
  11. ©ɹ2013 Internet Initiative Japan Inc. What is pmux? (1) • 

    stands for pipeline multiplexer •  written in Ruby • https://github.com/iij/pmux! • https://forge.gluster.org/pmux!
  12. ©ɹ2013 Internet Initiative Japan Inc. What is pmux? (2) • 

    file-based map/reduce tool •  uses Unix standard input/output as the interface $ pmux --mapper="grep PATTERN" *.log Example: distributed grep files on GlusterFS
  13. ©ɹ2013 Internet Initiative Japan Inc. What is pmux? (3)

  14. ©ɹ2013 Internet Initiative Japan Inc. Install $ gem install pmux

    $ gem install pmux $ gem install gflocator $ sudo gflocator
  15. ©ɹ2013 Internet Initiative Japan Inc. Execution Overview (1) MapReduce without

    reduce phase
  16. ©ɹ2013 Internet Initiative Japan Inc. 1. lookup target files run

    pmux command on this host read USVTUFEHMVTUFSGTQBUIJOGP from xattr
  17. ©ɹ2013 Internet Initiative Japan Inc. Extended file attributes (xattr) • 

    a file system feature that enables users to associate computer files with metadata (wikipedia) •  GlusterFS uses Extended Attributes as a mechanism for external interaction with translators.
  18. ©ɹ2013 Internet Initiative Japan Inc. Extended file attributes (2) $

    sudo getfattr -n trusted.glusterfs.pathinfo \! access_log.20131020! # file: access_log.20131020! trusted.glusterfs.pathinfo="(<DISTRIBUTE:d2r2-! dht> (<REPLICATE:d2r2-replicate-0> <POSIX(/glu! sterfs/brick/d2r2):ex01.example.com:/glusterfs! /brick/d2r2/log/0000/access_log.20131020> <POS! IX(/glusterfs/brick/d2r2):ex00.example.com:/gl! usterfs/brick/d2r2/log/0000/access_log.2013102! 0>))"
  19. ©ɹ2013 Internet Initiative Japan Inc. Extended file attributes (3) (glusterfs-hadoop

    GlusterFSXattr.java)
  20. ©ɹ2013 Internet Initiative Japan Inc. 2. invoke pmux on each

    node dispatcher worker
  21. ©ɹ2013 Internet Initiative Japan Inc. 3. assign map tasks to

    nodes tasks are assigned to nodes(workers) dynamically dispatcher worker
  22. ©ɹ2013 Internet Initiative Japan Inc. 4. popen (execute map task)

    dispatcher worker
  23. ©ɹ2013 Internet Initiative Japan Inc. 5. send results back to

    the dispatcher dispatcher worker
  24. ©ɹ2013 Internet Initiative Japan Inc. Execution Overview (2) with reduce

    phase
  25. ©ɹ2013 Internet Initiative Japan Inc. 4. popen (execute map task)

    dispatcher worker
  26. ©ɹ2013 Internet Initiative Japan Inc. 5. mapper produces tmp files

    mapper produces temporary files containing intermediate results dispatcher worker
  27. ©ɹ2013 Internet Initiative Japan Inc. 6. shuffle dispatcher worker

  28. ©ɹ2013 Internet Initiative Japan Inc. 7. assign reduce tasks to

    nodes dispatcher worker
  29. ©ɹ2013 Internet Initiative Japan Inc. 8. send results back to

    the dispatcher dispatcher worker
  30. ©ɹ2013 Internet Initiative Japan Inc. example(1): count of status code

    extract the status code from Apache log files and count $ pmux --mapper='cut -d" " -f 9’ \ --reducer='sort|uniq -c’ /mnt/glusterfs/*.log 176331 200 106360 206 809 400 21852 403 533 404 27 406 805 416 25 500
  31. ©ɹ2013 Internet Initiative Japan Inc. example(2): word count $ pmux

    --mapper=map.rb --reducer=reduce.rb \ --file=map.rb –-file=reduce.rb \ /mnt/glusterfs/*.txt #! /usr/bin/ruby -an $F.each {|f| print "#{f}\t1\n"} #! /usr/bin/ruby -an BEGIN {$c = Hash.new 0} $c[$F[0]] += $F[1].to_i END {$c.each {|k, v| print "#{k} #{v}\n"}} map.rb reduce.rb command line
  32. ©ɹ2013 Internet Initiative Japan Inc. Performance 14:00:00.416011 IP 21.44.60.29.http >

    170.73.162.175.58546: . 3523999974:3524001422(1448) ack 3401170238 win 1716 <nop,nop,timestamp 1070614671 1955062367> packet capture logs (made by tcpdump) like this extract the most frequently appeared IP address on each file 8344 files, 500K lines/file, total 4 billion lines
  33. ©ɹ2013 Internet Initiative Japan Inc. map command --mapper='egrep –o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"|

    sort|uniq -c|sort -nr|head -1'
  34. ©ɹ2013 Internet Initiative Japan Inc. result 8 hr 49 min

    6 sec 1 node, without pmux
  35. ©ɹ2013 Internet Initiative Japan Inc. result 8 hr 49 min

    6 sec 1 min 45 sec 300 times fater 1 node, without pmux 60 nodes (each node has 8 cores)