Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
glusterfs-pmux
maebashi
November 13, 2013
Technology
2
230
glusterfs-pmux
GlusterFSを利用した軽量MapReduceフレームワークpmux
2013/11/13 Gluster Cloud Night 発表資料
(於 レッドハット株式会社)
maebashi
November 13, 2013
Tweet
Share
More Decks by maebashi
See All by maebashi
docker-metrics-grafana
maebashi
2
840k
monitoring-tool-gri
maebashi
1
290
glusterfs-pmux-en
maebashi
0
26
pmux
maebashi
0
16
Other Decks in Technology
See All in Technology
LINE WORKS API 2.0について
mmclsntr
0
140
[SRE NEXT 2022]ヤプリのSREにおけるセキュリティ強化の取り組みを公開する
mmochi23
1
820
Poolにおける足を止めないシステム基盤構築
winebarrel
3
1.1k
[SRE NEXT 2022]組織に対してSREを適用するとはどういうことか
srenext
0
470
Puny to Powerful PostgreSQL Rails Apps
andyatkinson
PRO
0
400
toilを解消した話
asumaywy
0
210
長年運用されてきたモノリシックアプリケーションをコンテナ化しようとするとどんな問題に遭遇するか? / SRE NEXT 2022
nulabinc
PRO
15
8k
失敗を経験したあなたへ〜建設的なインシデントの振り返りを行うために実践するべきこと〜
nobuakikikuchi
0
210
プロダクトの理想と現実はなぜ乖離しがち?プロダクト作りに潜む問題を考える
suzukentaro
0
210
runn is a package/tool for running operations following a scenario. / golang.tokyo #32
k1low
1
240
220521_SFN_品質文化試論と『LEADING QUALITY』/220521_SFN_Essay_of_Quality_Culture_and_LEADING_QUALITY
mkwrd
0
310
A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
takmin
0
230
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
34
6.1k
Building Adaptive Systems
keathley
25
1.1k
Embracing the Ebb and Flow
colly
73
3.3k
Documentation Writing (for coders)
carmenhchung
48
2.5k
Adopting Sorbet at Scale
ufuk
63
7.5k
Side Projects
sachag
449
37k
Docker and Python
trallard
27
1.5k
Building Better People: How to give real-time feedback that sticks.
wjessup
343
17k
The Most Common Mistakes in Cover Letters
jrick
PRO
4
24k
Building Flexible Design Systems
yeseniaperezcruz
310
33k
Happy Clients
brianwarren
89
5.5k
Why Our Code Smells
bkeepers
PRO
324
54k
Transcript
©ɹ2013 Internet Initiative Japan Inc. ©ɹ2013 Internet Initiative Japan Inc.
GlusterFSΛར༻ͨ͠ ܰྔMapReduceϑϨʔϜϫʔΫ Pmux גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ maebashi@iij.ad.jp
©ɹ2013 Internet Initiative Japan Inc. ࣗݾհ • લڮ(Takahiro Maebashi) •
גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ(IIJ) • ITpro: ITݕূϥϘ -- ࢄϑΝΠϧγεςϜͷ GlusterFSɿ͜Μͳͱ͖ɺͲ͏ͳΔ – http://itpro.nikkeibp.co.jp/ article/COLUMN/20130104/447701/!
©ɹ2013 Internet Initiative Japan Inc. GlusterFS @ IIJ Tokyo Osaka
Matsue
©ɹ2013 Internet Initiative Japan Inc. ίϯςφϢχοτʮIZmoʯ (Matsue Data Center Park)
IT module air-conditioning unit
©ɹ2013 Internet Initiative Japan Inc. GlusterFS servers in IZmo •
ϥοΫ͕ࣼΊʹஔ͞Ε͍ͯΔ
©ɹ2013 Internet Initiative Japan Inc. Today's Talk
©ɹ2013 Internet Initiative Japan Inc. glusterfs-hadoop
©ɹ2013 Internet Initiative Japan Inc. What is MapReduce? MapͱReduceͷ2ஈ֊Ͱࢄॲཧ (1)
Map – நग़ɺม (2) Reduce – ूɺूܭ
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS?
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS? (2) (ྫ:
distributed volume ͷ߹) ϑΝΠϧ୯ҐͰɺϑΝΠϧ໊ʹԠͯ͡ࢄ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (1) •
pipeline multiplexer ʹ༝དྷ • RubyͰهड़͞Ε͍ͯΔ • https://github.com/iij/pmux! • https://forge.gluster.org/pmux!
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (2) •
ϑΝΠϧϕʔεͷ map/reduce πʔϧ • Unix ͷඪ४ೖྗ/ग़ྗΛΠϯλϑΣʔεͱͯ͠ ͏ $ pmux --mapper="grep PATTERN" *.log Example: ࢄgrep GlusterFS্ͷϑΝΠϧ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (3)
©ɹ2013 Internet Initiative Japan Inc. Install $ gem install pmux
$ gem install pmux $ gem install gflocator $ sudo gflocator
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (1) MapReduceɺreduce phaseͳ͠ͷ߹
©ɹ2013 Internet Initiative Japan Inc. 1. ରͱ͢ΔϑΝΠϧ܈Λ୳͢ pmux ίϚϯυ͜ͷϗετ Ͱ࣮ߦ͢Δ
USVTUFEHMVTUFSGTQBUIJOGP ΛಡΈग़͢
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ(xattr) • ϝλσʔλΛϢʔβ͕ϑΝΠϧʹ݁ͼ͚ͭΔ͜ ͱ͕ग़དྷΔΑ͏ʹ͢ΔϑΝΠϧγεςϜͷػ ೳ
(wikipedia) • GlusterFS ɺ֦ுϑΝΠϧଐੑΛ֎෦ͱΓ ͱΓ͢ΔͨΊͷΈͱͯ͠͏
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (2) $ sudo getfattr
-n trusted.glusterfs.pathinfo \! access_log.20131020! # file: access_log.20131020! trusted.glusterfs.pathinfo="(<DISTRIBUTE:d2r2-! dht> (<REPLICATE:d2r2-replicate-0> <POSIX(/glu! sterfs/brick/d2r2):ex01.example.com:/glusterfs! /brick/d2r2/log/0000/access_log.20131020> <POS! IX(/glusterfs/brick/d2r2):ex00.example.com:/gl! usterfs/brick/d2r2/log/0000/access_log.2013102! 0>))"
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (3) (glusterfs-hadoop GlusterFSXattr.java)
©ɹ2013 Internet Initiative Japan Inc. 2. ֤ϊʔυͰpmuxΛىಈ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 3. map tasks Λ֤ϊʔυʹׂΓͯ tasks
are assigned to nodes(workers) dynamically dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. ݁ՌΛ dispatcher ʹฦ͢ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (2) reduce phase
͕͋Δ߹
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. mapper ͕Ұ࣌ϑΝΠϧੜ mapperதؒ݁ՌΛؚΜͩҰ࣌ϑΝΠϧΛੜ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 6. shuffle dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 7. reduce tasks ΛϊʔυʹׂΓͯ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 8. dispatcher ʹ݁ՌΛฦ͢ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. example(1): εςʔλείʔυΛ͑Δ Apache log ͷHTTPεςʔλείʔυͷग़ݱΛ͑Δ
$ pmux --mapper='cut -d" " -f 9’ \ --reducer='sort|uniq -c’ /mnt/glusterfs/*.log 176331 200 106360 206 809 400 21852 403 533 404 27 406 805 416 25 500
©ɹ2013 Internet Initiative Japan Inc. example(2): word count $ pmux
--mapper=map.rb --reducer=reduce.rb \ --file=map.rb –-file=reduce.rb \ /mnt/glusterfs/*.txt #! /usr/bin/ruby -an $F.each {|f| print "#{f}\t1\n"} #! /usr/bin/ruby -an BEGIN {$c = Hash.new 0} $c[$F[0]] += $F[1].to_i END {$c.each {|k, v| print "#{k} #{v}\n"}} map.rb reduce.rb command line
©ɹ2013 Internet Initiative Japan Inc. ੑೳ 14:00:00.416011 IP 21.44.60.29.http >
170.73.162.175.58546: . 3523999974:3524001422(1448) ack 3401170238 win 1716 <nop,nop,timestamp 1070614671 1955062367> ҎԼͷΑ͏ͳύέοτΩϟϓνϟϩά (by tcpdump) ֤ϑΝΠϧͰ࠷ग़ݱͷଟ͍IPΞυϨεΛநग़͢Δ 8344 files, 500K lines/file, total 4 billion lines
©ɹ2013 Internet Initiative Japan Inc. map ίϚϯυ --mapper='egrep –o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"|
sort|uniq -c|sort -nr|head -1'
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 node, without pmux
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 min 45 sec 300ഒ! 1 node, without pmux 60 nodes (֤ϊʔυ8ίΞ)