Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
glusterfs-pmux
Search
maebashi
November 13, 2013
Technology
2
550
glusterfs-pmux
GlusterFSを利用した軽量MapReduceフレームワークpmux
2013/11/13 Gluster Cloud Night 発表資料
(於 レッドハット株式会社)
maebashi
November 13, 2013
Tweet
Share
More Decks by maebashi
See All by maebashi
docker-metrics-grafana
maebashi
2
840k
monitoring-tool-gri
maebashi
1
500
glusterfs-pmux-en
maebashi
0
61
pmux
maebashi
0
79
Other Decks in Technology
See All in Technology
改めてAWS WAFを振り返る~業務で使うためのポイント~
masakiokuda
2
270
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
320
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
2
8k
[ JAWS-UG千葉支部 x 彩の国埼玉支部 ]ムダ遣い卒業!FinOpsで始めるAWSコスト最適化の第一歩
sh_fk2
2
100
20250705 Headlamp: 專注可擴展性的 Kubernetes 用戶界面
pichuang
0
280
データ基盤からデータベースまで?広がるユースケースのDatabricksについて教えるよ!
akuwano
3
120
Model Mondays S2E04: AI Developer Experiences
nitya
0
190
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
53
20k
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
2k
生まれ変わった AWS Security Hub (Preview) を紹介 #reInforce_osaka / reInforce New Security Hub
masahirokawahara
0
480
OSSのSNSツール「Misskey」をさわってみよう(右下ワイプで私のOSCの20年を振り返ります) / 20250705-osc2025-do
akkiesoft
0
170
赤煉瓦倉庫勉強会「Databricksを選んだ理由と、絶賛真っ只中のデータ基盤移行体験記」
ivry_presentationmaterials
2
370
Featured
See All Featured
GitHub's CSS Performance
jonrohan
1031
460k
How to Ace a Technical Interview
jacobian
278
23k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Scaling GitHub
holman
460
140k
The Cult of Friendly URLs
andyhume
79
6.5k
How GitHub (no longer) Works
holman
314
140k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Designing for humans not robots
tammielis
253
25k
Side Projects
sachag
455
42k
KATA
mclloyd
30
14k
Transcript
©ɹ2013 Internet Initiative Japan Inc. ©ɹ2013 Internet Initiative Japan Inc.
GlusterFSΛར༻ͨ͠ ܰྔMapReduceϑϨʔϜϫʔΫ Pmux גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ maebashi@iij.ad.jp
©ɹ2013 Internet Initiative Japan Inc. ࣗݾհ • લڮ(Takahiro Maebashi) •
גࣜձࣾΠϯλʔωοτΠχγΞςΟϒ(IIJ) • ITpro: ITݕূϥϘ -- ࢄϑΝΠϧγεςϜͷ GlusterFSɿ͜Μͳͱ͖ɺͲ͏ͳΔ – http://itpro.nikkeibp.co.jp/ article/COLUMN/20130104/447701/!
©ɹ2013 Internet Initiative Japan Inc. GlusterFS @ IIJ Tokyo Osaka
Matsue
©ɹ2013 Internet Initiative Japan Inc. ίϯςφϢχοτʮIZmoʯ (Matsue Data Center Park)
IT module air-conditioning unit
©ɹ2013 Internet Initiative Japan Inc. GlusterFS servers in IZmo •
ϥοΫ͕ࣼΊʹஔ͞Ε͍ͯΔ
©ɹ2013 Internet Initiative Japan Inc. Today's Talk
©ɹ2013 Internet Initiative Japan Inc. glusterfs-hadoop
©ɹ2013 Internet Initiative Japan Inc. What is MapReduce? MapͱReduceͷ2ஈ֊Ͱࢄॲཧ (1)
Map – நग़ɺม (2) Reduce – ूɺूܭ
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS?
©ɹ2013 Internet Initiative Japan Inc. What is GlusterFS? (2) (ྫ:
distributed volume ͷ߹) ϑΝΠϧ୯ҐͰɺϑΝΠϧ໊ʹԠͯ͡ࢄ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (1) •
pipeline multiplexer ʹ༝དྷ • RubyͰهड़͞Ε͍ͯΔ • https://github.com/iij/pmux! • https://forge.gluster.org/pmux!
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (2) •
ϑΝΠϧϕʔεͷ map/reduce πʔϧ • Unix ͷඪ४ೖྗ/ग़ྗΛΠϯλϑΣʔεͱͯ͠ ͏ $ pmux --mapper="grep PATTERN" *.log Example: ࢄgrep GlusterFS্ͷϑΝΠϧ
©ɹ2013 Internet Initiative Japan Inc. What is pmux? (3)
©ɹ2013 Internet Initiative Japan Inc. Install $ gem install pmux
$ gem install pmux $ gem install gflocator $ sudo gflocator
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (1) MapReduceɺreduce phaseͳ͠ͷ߹
©ɹ2013 Internet Initiative Japan Inc. 1. ରͱ͢ΔϑΝΠϧ܈Λ୳͢ pmux ίϚϯυ͜ͷϗετ Ͱ࣮ߦ͢Δ
USVTUFEHMVTUFSGTQBUIJOGP ΛಡΈग़͢
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ(xattr) • ϝλσʔλΛϢʔβ͕ϑΝΠϧʹ݁ͼ͚ͭΔ͜ ͱ͕ग़དྷΔΑ͏ʹ͢ΔϑΝΠϧγεςϜͷػ ೳ
(wikipedia) • GlusterFS ɺ֦ுϑΝΠϧଐੑΛ֎෦ͱΓ ͱΓ͢ΔͨΊͷΈͱͯ͠͏
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (2) $ sudo getfattr
-n trusted.glusterfs.pathinfo \! access_log.20131020! # file: access_log.20131020! trusted.glusterfs.pathinfo="(<DISTRIBUTE:d2r2-! dht> (<REPLICATE:d2r2-replicate-0> <POSIX(/glu! sterfs/brick/d2r2):ex01.example.com:/glusterfs! /brick/d2r2/log/0000/access_log.20131020> <POS! IX(/glusterfs/brick/d2r2):ex00.example.com:/gl! usterfs/brick/d2r2/log/0000/access_log.2013102! 0>))"
©ɹ2013 Internet Initiative Japan Inc. ֦ுϑΝΠϧଐੑ (3) (glusterfs-hadoop GlusterFSXattr.java)
©ɹ2013 Internet Initiative Japan Inc. 2. ֤ϊʔυͰpmuxΛىಈ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 3. map tasks Λ֤ϊʔυʹׂΓͯ tasks
are assigned to nodes(workers) dynamically dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. ݁ՌΛ dispatcher ʹฦ͢ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. Execution Overview (2) reduce phase
͕͋Δ߹
©ɹ2013 Internet Initiative Japan Inc. 4. popen (map task ࣮ߦ)
dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 5. mapper ͕Ұ࣌ϑΝΠϧੜ mapperதؒ݁ՌΛؚΜͩҰ࣌ϑΝΠϧΛੜ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 6. shuffle dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. 7. reduce tasks ΛϊʔυʹׂΓͯ dispatcher
worker
©ɹ2013 Internet Initiative Japan Inc. 8. dispatcher ʹ݁ՌΛฦ͢ dispatcher worker
©ɹ2013 Internet Initiative Japan Inc. example(1): εςʔλείʔυΛ͑Δ Apache log ͷHTTPεςʔλείʔυͷग़ݱΛ͑Δ
$ pmux --mapper='cut -d" " -f 9’ \ --reducer='sort|uniq -c’ /mnt/glusterfs/*.log 176331 200 106360 206 809 400 21852 403 533 404 27 406 805 416 25 500
©ɹ2013 Internet Initiative Japan Inc. example(2): word count $ pmux
--mapper=map.rb --reducer=reduce.rb \ --file=map.rb –-file=reduce.rb \ /mnt/glusterfs/*.txt #! /usr/bin/ruby -an $F.each {|f| print "#{f}\t1\n"} #! /usr/bin/ruby -an BEGIN {$c = Hash.new 0} $c[$F[0]] += $F[1].to_i END {$c.each {|k, v| print "#{k} #{v}\n"}} map.rb reduce.rb command line
©ɹ2013 Internet Initiative Japan Inc. ੑೳ 14:00:00.416011 IP 21.44.60.29.http >
170.73.162.175.58546: . 3523999974:3524001422(1448) ack 3401170238 win 1716 <nop,nop,timestamp 1070614671 1955062367> ҎԼͷΑ͏ͳύέοτΩϟϓνϟϩά (by tcpdump) ֤ϑΝΠϧͰ࠷ग़ݱͷଟ͍IPΞυϨεΛநग़͢Δ 8344 files, 500K lines/file, total 4 billion lines
©ɹ2013 Internet Initiative Japan Inc. map ίϚϯυ --mapper='egrep –o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+"|
sort|uniq -c|sort -nr|head -1'
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 node, without pmux
©ɹ2013 Internet Initiative Japan Inc. ݁Ռ 8 hr 49 min
6 sec 1 min 45 sec 300ഒ! 1 node, without pmux 60 nodes (֤ϊʔυ8ίΞ)