Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
glusterfs-pmux
Search
maebashi
November 13, 2013
Technology
2
570
glusterfs-pmux
GlusterFSを利用した軽量MapReduceフレームワークpmux
2013/11/13 Gluster Cloud Night 発表資料
(於 レッドハット株式会社)
maebashi
November 13, 2013
Tweet
Share
More Decks by maebashi
See All by maebashi
docker-metrics-grafana
maebashi
2
840k
monitoring-tool-gri
maebashi
1
510
glusterfs-pmux-en
maebashi
0
67
pmux
maebashi
0
84
Other Decks in Technology
See All in Technology
react-callを使ってダイヤログをいろんなとこで再利用しよう!
shinaps
2
260
プラットフォーム転換期におけるGitHub Copilot活用〜Coding agentがそれを加速するか〜 / Leveraging GitHub Copilot During Platform Transition Periods
aeonpeople
1
210
品質視点から考える組織デザイン/Organizational Design from Quality
mii3king
0
210
Agile PBL at New Grads Trainings
kawaguti
PRO
1
450
AWSを利用する上で知っておきたい名前解決のはなし(10分版)
nagisa53
10
3.2k
スマートファクトリーの第一歩 〜AWSマネージドサービスで 実現する予知保全と生成AI活用まで
ganota
2
290
普通のチームがスクラムを会得するたった一つの冴えたやり方 / the best way to scrum
okamototakuyasr2
0
110
共有と分離 - Compose Multiplatform "本番導入" の設計指針
error96num
2
1k
下手な強制、ダメ!絶対! 「ガードレール」を「檻」にさせない"ガバナンス"の取り方とは?
tsukaman
2
450
2つのフロントエンドと状態管理
mixi_engineers
PRO
3
110
要件定義・デザインフェーズでもAIを活用して、コミュニケーションの密度を高める
kazukihayase
0
120
研究開発と製品開発、両利きのロボティクス
youtalk
1
530
Featured
See All Featured
Fireside Chat
paigeccino
39
3.6k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Context Engineering - Making Every Token Count
addyosmani
3
58
Bash Introduction
62gerente
615
210k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
How STYLIGHT went responsive
nonsquared
100
5.8k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
How to Think Like a Performance Engineer
csswizardry
26
1.9k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
Transcript
©ɹ2013 Internet Initiative Japan Inc. ©ɹ2013 Internet Initiative Japan Inc.
GlusterFSΛར༻ͨ͠ ܰྔMapReduceϑϨʔϜϫʔΫ Pmux גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ
[email protected]
©ɹ2013 Internet Initiative Japan Inc. ࣗݾհ • લڮ(Takahiro Maebashi) •
גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ(IIJ) • ITpro: ITݕূϥϘ -- ࢄϑΝΠϧγεςϜͷ GlusterFSɿ͜Μͳͱ͖ɺͲ͏ͳΔ – http://itpro.nikkeibp.co.jp/ article/COLUMN/20130104/447701/!
©ɹ2013 Internet Initiative Japan Inc. GlusterFS @ IIJ Tokyo Osaka
Matsue
©ɹ2013 Internet Initiative Japan Inc. ίϯςφϢχοτʮIZmoʯ (Matsue Data Center Park)
IT module air-conditioning unit
©ɹ2013 Internet Initiative Japan Inc. GlusterFS servers in IZmo •
ϥοΫ͕ࣼΊʹஔ͞Ε͍ͯΔ
©ɹ2013 Internet Initiative Japan Inc. Today's Talk
©ɹ2013 Internet Initiative Japan Inc. glusterfs-hadoop
©ɹ2013 Internet Initiative Japan Inc. What is MapReduce? MapͱReduceͷ2ஈ֊Ͱࢄॲཧ (1)
Map – நग़ɺม (2) Reduce – ूɺूܭ
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS?
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS? (2) (ྫ:
distributed volume ͷ߹) ϑΝΠϧ୯ҐͰɺϑΝΠϧ໊ʹԠͯ͡ࢄ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (1) •
pipeline multiplexer ʹ༝དྷ • RubyͰهड़͞Ε͍ͯΔ • https://github.com/iij/pmux! • https://forge.gluster.org/pmux!
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (2) •
ϑΝΠϧϕʔεͷ map/reduce πʔϧ • Unix ͷඪ४ೖྗ/ग़ྗΛΠϯλϑΣʔεͱͯ͠ ͏ $ pmux --mapper="grep PATTERN" *.log Example: ࢄgrep GlusterFS্ͷϑΝΠϧ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (3)
©ɹ2013 Internet Initiative Japan Inc. Install $ gem install pmux
$ gem install pmux $ gem install gflocator $ sudo gflocator
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (1) MapReduceɺreduce phaseͳ͠ͷ߹
©ɹ2013 Internet Initiative Japan Inc. 1. ରͱ͢ΔϑΝΠϧ܈Λ୳͢ pmux ίϚϯυ͜ͷϗετ Ͱ࣮ߦ͢Δ
USVTUFEHMVTUFSGTQBUIJOGP ΛಡΈग़͢
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ(xattr) • ϝλσʔλΛϢʔβ͕ϑΝΠϧʹ݁ͼ͚ͭΔ͜ ͱ͕ग़དྷΔΑ͏ʹ͢ΔϑΝΠϧγεςϜͷػ ೳ
(wikipedia) • GlusterFS ɺ֦ுϑΝΠϧଐੑΛ֎෦ͱΓ ͱΓ͢ΔͨΊͷΈͱͯ͠͏
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (2) $ sudo getfattr
-n trusted.glusterfs.pathinfo \! access_log.20131020! # file: access_log.20131020! trusted.glusterfs.pathinfo="(<DISTRIBUTE:d2r2-! dht> (<REPLICATE:d2r2-replicate-0> <POSIX(/glu! sterfs/brick/d2r2):ex01.example.com:/glusterfs! /brick/d2r2/log/0000/access_log.20131020> <POS! IX(/glusterfs/brick/d2r2):ex00.example.com:/gl! usterfs/brick/d2r2/log/0000/access_log.2013102! 0>))"
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (3) (glusterfs-hadoop GlusterFSXattr.java)
©ɹ2013 Internet Initiative Japan Inc. 2. ֤ϊʔυͰpmuxΛىಈ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 3. map tasks Λ֤ϊʔυʹׂΓͯ tasks
are assigned to nodes(workers) dynamically dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. ݁ՌΛ dispatcher ʹฦ͢ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (2) reduce phase
͕͋Δ߹
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. mapper ͕Ұ࣌ϑΝΠϧੜ mapperதؒ݁ՌΛؚΜͩҰ࣌ϑΝΠϧΛੜ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 6. shuffle dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 7. reduce tasks ΛϊʔυʹׂΓͯ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 8. dispatcher ʹ݁ՌΛฦ͢ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. example(1): εςʔλείʔυΛ͑Δ Apache log ͷHTTPεςʔλείʔυͷग़ݱΛ͑Δ
$ pmux --mapper='cut -d" " -f 9’ \ --reducer='sort|uniq -c’ /mnt/glusterfs/*.log 176331 200 106360 206 809 400 21852 403 533 404 27 406 805 416 25 500
©ɹ2013 Internet Initiative Japan Inc. example(2): word count $ pmux
--mapper=map.rb --reducer=reduce.rb \ --file=map.rb –-file=reduce.rb \ /mnt/glusterfs/*.txt #! /usr/bin/ruby -an $F.each {|f| print "#{f}\t1\n"} #! /usr/bin/ruby -an BEGIN {$c = Hash.new 0} $c[$F[0]] += $F[1].to_i END {$c.each {|k, v| print "#{k} #{v}\n"}} map.rb reduce.rb command line
©ɹ2013 Internet Initiative Japan Inc. ੑೳ 14:00:00.416011 IP 21.44.60.29.http >
170.73.162.175.58546: . 3523999974:3524001422(1448) ack 3401170238 win 1716 <nop,nop,timestamp 1070614671 1955062367> ҎԼͷΑ͏ͳύέοτΩϟϓνϟϩά (by tcpdump) ֤ϑΝΠϧͰ࠷ग़ݱͷଟ͍IPΞυϨεΛநग़͢Δ 8344 files, 500K lines/file, total 4 billion lines
©ɹ2013 Internet Initiative Japan Inc. map ίϚϯυ --mapper='egrep –o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"|
sort|uniq -c|sort -nr|head -1'
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 node, without pmux
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 min 45 sec 300ഒ! 1 node, without pmux 60 nodes (֤ϊʔυ8ίΞ)