Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
glusterfs-pmux
Search
maebashi
November 13, 2013
Technology
2
550
glusterfs-pmux
GlusterFSを利用した軽量MapReduceフレームワークpmux
2013/11/13 Gluster Cloud Night 発表資料
(於 レッドハット株式会社)
maebashi
November 13, 2013
Tweet
Share
More Decks by maebashi
See All by maebashi
docker-metrics-grafana
maebashi
2
840k
monitoring-tool-gri
maebashi
1
500
glusterfs-pmux-en
maebashi
0
60
pmux
maebashi
0
79
Other Decks in Technology
See All in Technology
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
26k
OpenHands🤲にContributeしてみた
kotauchisunsun
1
480
米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
2
1.2k
PHP開発者のためのSOLID原則再入門 #phpcon / PHP Conference Japan 2025
shogogg
4
900
Liquid Glass革新とSwiftUI/UIKit進化
fumiyasac0921
0
280
Core Audio tapを使ったリアルタイム音声処理のお話
yuta0306
0
120
Should Our Project Join the CNCF? (Japanese Recap)
whywaita
PRO
0
270
250627 関西Ruby会議08 前夜祭 RejectKaigi「DJ on Ruby Ver.0.1」
msykd
PRO
2
350
Snowflake Summit 2025全体振り返り / Snowflake Summit 2025 Overall Review
mtpooh
2
420
プロダクトエンジニアリング組織への歩み、その現在地 / Our journey to becoming a product engineering organization
hiro_torii
0
130
データプラットフォーム技術におけるメダリオンアーキテクチャという考え方/DataPlatformWithMedallionArchitecture
smdmts
5
660
PHPでWebブラウザのレンダリングエンジンを実装する
dip_tech
PRO
0
210
Featured
See All Featured
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
BBQ
matthewcrist
89
9.7k
Stop Working from a Prison Cell
hatefulcrawdad
270
20k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Statistics for Hackers
jakevdp
799
220k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Done Done
chrislema
184
16k
Why Our Code Smells
bkeepers
PRO
337
57k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Transcript
©ɹ2013 Internet Initiative Japan Inc. ©ɹ2013 Internet Initiative Japan Inc.
GlusterFSΛར༻ͨ͠ ܰྔMapReduceϑϨʔϜϫʔΫ Pmux גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ
[email protected]
©ɹ2013 Internet Initiative Japan Inc. ࣗݾհ • લڮ(Takahiro Maebashi) •
גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ(IIJ) • ITpro: ITݕূϥϘ -- ࢄϑΝΠϧγεςϜͷ GlusterFSɿ͜Μͳͱ͖ɺͲ͏ͳΔ – http://itpro.nikkeibp.co.jp/ article/COLUMN/20130104/447701/!
©ɹ2013 Internet Initiative Japan Inc. GlusterFS @ IIJ Tokyo Osaka
Matsue
©ɹ2013 Internet Initiative Japan Inc. ίϯςφϢχοτʮIZmoʯ (Matsue Data Center Park)
IT module air-conditioning unit
©ɹ2013 Internet Initiative Japan Inc. GlusterFS servers in IZmo •
ϥοΫ͕ࣼΊʹஔ͞Ε͍ͯΔ
©ɹ2013 Internet Initiative Japan Inc. Today's Talk
©ɹ2013 Internet Initiative Japan Inc. glusterfs-hadoop
©ɹ2013 Internet Initiative Japan Inc. What is MapReduce? MapͱReduceͷ2ஈ֊Ͱࢄॲཧ (1)
Map – நग़ɺม (2) Reduce – ूɺूܭ
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS?
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS? (2) (ྫ:
distributed volume ͷ߹) ϑΝΠϧ୯ҐͰɺϑΝΠϧ໊ʹԠͯ͡ࢄ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (1) •
pipeline multiplexer ʹ༝དྷ • RubyͰهड़͞Ε͍ͯΔ • https://github.com/iij/pmux! • https://forge.gluster.org/pmux!
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (2) •
ϑΝΠϧϕʔεͷ map/reduce πʔϧ • Unix ͷඪ४ೖྗ/ग़ྗΛΠϯλϑΣʔεͱͯ͠ ͏ $ pmux --mapper="grep PATTERN" *.log Example: ࢄgrep GlusterFS্ͷϑΝΠϧ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (3)
©ɹ2013 Internet Initiative Japan Inc. Install $ gem install pmux
$ gem install pmux $ gem install gflocator $ sudo gflocator
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (1) MapReduceɺreduce phaseͳ͠ͷ߹
©ɹ2013 Internet Initiative Japan Inc. 1. ରͱ͢ΔϑΝΠϧ܈Λ୳͢ pmux ίϚϯυ͜ͷϗετ Ͱ࣮ߦ͢Δ
USVTUFEHMVTUFSGTQBUIJOGP ΛಡΈग़͢
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ(xattr) • ϝλσʔλΛϢʔβ͕ϑΝΠϧʹ݁ͼ͚ͭΔ͜ ͱ͕ग़དྷΔΑ͏ʹ͢ΔϑΝΠϧγεςϜͷػ ೳ
(wikipedia) • GlusterFS ɺ֦ுϑΝΠϧଐੑΛ֎෦ͱΓ ͱΓ͢ΔͨΊͷΈͱͯ͠͏
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (2) $ sudo getfattr
-n trusted.glusterfs.pathinfo \! access_log.20131020! # file: access_log.20131020! trusted.glusterfs.pathinfo="(<DISTRIBUTE:d2r2-! dht> (<REPLICATE:d2r2-replicate-0> <POSIX(/glu! sterfs/brick/d2r2):ex01.example.com:/glusterfs! /brick/d2r2/log/0000/access_log.20131020> <POS! IX(/glusterfs/brick/d2r2):ex00.example.com:/gl! usterfs/brick/d2r2/log/0000/access_log.2013102! 0>))"
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (3) (glusterfs-hadoop GlusterFSXattr.java)
©ɹ2013 Internet Initiative Japan Inc. 2. ֤ϊʔυͰpmuxΛىಈ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 3. map tasks Λ֤ϊʔυʹׂΓͯ tasks
are assigned to nodes(workers) dynamically dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. ݁ՌΛ dispatcher ʹฦ͢ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (2) reduce phase
͕͋Δ߹
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. mapper ͕Ұ࣌ϑΝΠϧੜ mapperதؒ݁ՌΛؚΜͩҰ࣌ϑΝΠϧΛੜ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 6. shuffle dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 7. reduce tasks ΛϊʔυʹׂΓͯ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 8. dispatcher ʹ݁ՌΛฦ͢ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. example(1): εςʔλείʔυΛ͑Δ Apache log ͷHTTPεςʔλείʔυͷग़ݱΛ͑Δ
$ pmux --mapper='cut -d" " -f 9’ \ --reducer='sort|uniq -c’ /mnt/glusterfs/*.log 176331 200 106360 206 809 400 21852 403 533 404 27 406 805 416 25 500
©ɹ2013 Internet Initiative Japan Inc. example(2): word count $ pmux
--mapper=map.rb --reducer=reduce.rb \ --file=map.rb –-file=reduce.rb \ /mnt/glusterfs/*.txt #! /usr/bin/ruby -an $F.each {|f| print "#{f}\t1\n"} #! /usr/bin/ruby -an BEGIN {$c = Hash.new 0} $c[$F[0]] += $F[1].to_i END {$c.each {|k, v| print "#{k} #{v}\n"}} map.rb reduce.rb command line
©ɹ2013 Internet Initiative Japan Inc. ੑೳ 14:00:00.416011 IP 21.44.60.29.http >
170.73.162.175.58546: . 3523999974:3524001422(1448) ack 3401170238 win 1716 <nop,nop,timestamp 1070614671 1955062367> ҎԼͷΑ͏ͳύέοτΩϟϓνϟϩά (by tcpdump) ֤ϑΝΠϧͰ࠷ग़ݱͷଟ͍IPΞυϨεΛநग़͢Δ 8344 files, 500K lines/file, total 4 billion lines
©ɹ2013 Internet Initiative Japan Inc. map ίϚϯυ --mapper='egrep –o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"|
sort|uniq -c|sort -nr|head -1'
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 node, without pmux
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 min 45 sec 300ഒ! 1 node, without pmux 60 nodes (֤ϊʔυ8ίΞ)