Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
dsync: Efficient Block-wise Synchronization of ...
Search
Yuuki Tsubouchi (yuuk1)
May 23, 2014
Technology
2
26k
dsync: Efficient Block-wise Synchronization of Multi-Gigabyte Binary Data
論文輪読会#4
ブロックデバイスレベルで実現するrsyncより高速なバックアップについて
Yuuki Tsubouchi (yuuk1)
May 23, 2014
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
1
25
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.7k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
190
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
13k
Cloudless Computingの論文紹介
yuukit
2
490
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
1.8k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
280
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
3k
Other Decks in Technology
See All in Technology
Aurora PostgreSQLがCloudWatch Logsに 出力するログの課金を削減してみる #jawsdays2025
non97
1
210
サイト信頼性エンジニアリングとAmazon Web Services / SRE and AWS
ymotongpoo
7
1.6k
Active Directory攻防
cryptopeg
PRO
8
5.5k
JavaにおけるNull非許容性
skrb
2
2.6k
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
2
1.5k
Iceberg Meetup Japan #1 : Iceberg and Databricks
databricksjapan
0
370
Pwned Labsのすゝめ
ken5scal
2
440
データエンジニアリング領域におけるDuckDBのユースケース
chanyou0311
9
2.2k
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
220
Windows の新しい管理者保護モード
murachiakira
0
200
IAMのマニアックな話2025
nrinetcom
PRO
4
650
ウォンテッドリーのデータパイプラインを支える ETL のための analytics, rds-exporter / analytics, rds-exporter for ETL to support Wantedly's data pipeline
unblee
0
130
Featured
See All Featured
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
How to Ace a Technical Interview
jacobian
276
23k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.4k
Writing Fast Ruby
sferik
628
61k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
BBQ
matthewcrist
87
9.5k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Optimizing for Happiness
mojombo
376
70k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
How STYLIGHT went responsive
nonsquared
98
5.4k
Docker and Python
trallard
44
3.3k
Transcript
dsync: Efficient Block-wise Synchronization of Multi Gigabyte Binary data Thomas
Knauth and Christof Fetzer, Technische Universität Dresden ! LISA’13 Best Paper Award id:y_uuki 2014/05/22 จྠಡձ#4
Agenda Backup Problem Implementation Device mapper Evaluation Discussion Conclusion
Better Backup • ωοτϫʔΫτϥϑΟοΫͷ࠷খԽ • νΣοΫαϜܭࢉͷCPUίετ͕ͳ͍ • σΟεΫͷread/writeͷ࠷খԽ • OSͷϖʔδΩϟογϡԚછͷ࠷খԽ
Backup ͷલఏ • ఆظతʹͳΔ͘සൟʹόοΫΞοϓ͍ͨ͠ • σʔλࣗମʹ΄ͱΜͲมߋ͕ͳ͍ • શσʔλྔʹରͯ͠มߋͷ͋ͬͨσʔλ ͔ᷮ
Problem • nc/scp: શσʔλίϐʔ • ωοτϫʔΫτϥϑΟοΫେ • 10Gbps Ethernet,100GB, 83
sec (1.2GB/s) • rsync: ͚ࠩͩసૹ • ࠩܭࢉͷͨΊʹશσʔλಡΈग़͠ඞཁ • νΣοΫαϜܭࢉͷCPUίετߴ͍ • OSͷΩϟογϡԚછ
rsync ᶃ ૹ৴ଆͱड৴ଆͷಉظ͍ͨ͠ϑΝΠϧΛݻఆϒϩοΫʹׂ ! ϒϩοΫ୯ҐͰࠩΛௐͯɺࠩͷ͋ΔϒϩοΫ͚ͩసૹ͍ͨ͠ ! ᶄ ֤ϒϩοΫʹରͯ͠νΣοΫαϜΛܭࢉ͠ɺϒϩοΫͷ༰ͷΘ ΓʹνΣοΫαϜ͚ͩΛૹ৴ !
ᶅ νΣοΫαϜΛൺֱ͕ͯࠩ͋͠Δ͔Ͳ͏͔ΛνΣοΫ ऑ͍νΣοΫαϜ(ܭࢉίετ)ͱڧ͍νΣοΫαϜ(ܭࢉίετߴ) ऑ͍νΣοΫαϜʢϩʔϦϯάνΣοΫαϜʣͰࠩͷ͋ΔϒϩοΫΛ ચ͍ग़ͯ͠ɺڧ͍νΣοΫαϜͰ࣮֬ʹࠩνΣοΫ
Idea • όοΫΞοϓ࣌ʹมߋՕॴΛܾఆ… • νΣοΫαϜΛൺֱ͢Δ͔͠ํ๏͕ͳ͍ • ࠷ॳ͔ΒมߋՕॴΛτϥοΩϯά͢Δ
Implementation • ϒϩοΫσόΠεϨϕϧͰมߋ͞Εͨϒϩο ΫͷτϥοΩϯάใΛอ࣋ • τϥοΩϯάใ: ϒϩοΫ͝ͱͷมߋ༗ແͷ ϑϥά ʢ1bit /
blockʣ • ߹ܭσʔλ: 4TiB -> ϑϥά: 128 MiB
Interface • ϢʔβεϖʔεΠϯλϑΣʔε • ֤σόΠε͝ͱʹ /proc ҎԼʹରԠ͢ΔϑΝΠϧ͕Ͱ͖Δ • /proc/mydev: ϒϩοΫ൪߸ͷϦετɻ͜Εʹॻ͖ࠐΉͱ
bit vector ॳظԽ͞ΕΔ • dmextract: มߋͷ͋ͬͨϒϩοΫͷநग़ stdout:ʢϒϩο Ϋ൪߸, σʔλʣ • dmextract mydev | ssh remotehost dmmerge /dev/ mapper/mydev
Block Device ϒϩοΫσόΠευϥΠό ετϨʔδσόΠε(HDD/SSD) ൚༻ϒϩοΫσόΠευϥΠό ΞϓϦέʔγϣϯ ϖʔδΩϟογϡ ϑΝΠϧγεςϜ ϑΝΠϧͷಡΈॻ͖ཁٻ ϑΝΠϧͱσόΠεͷϒϩοΫͱͷϚοϐϯά
*0ཁٻΛσόΠεʹదͨ͠ܗʹฒସ͑ͳͲ 3".ʹಡΈॻ͖σʔλΛΩϟογϡ ݸʑͷϋʔυΣΞʹ͋Θͤͨॲཧ ε τ Ϩ c δ σ ό Π ε ந Խ
Device mapper http://lc.linux.or.jp/lc2009/slide/T-02-slide.pdf ϒϩοΫσόΠευϥΠό ετϨʔδσόΠε(HDD/SSD) ൚༻ϒϩοΫσόΠευϥΠό ΞϓϦέʔγϣϯ ϖʔδΩϟογϡ ϑΝΠϧγεςϜ ϑΝΠϧͷಡΈॻ͖ཁٻ
ϑΝΠϧͱσόΠεͷϒϩοΫͱͷϚοϐϯά *0ཁٻΛσόΠεʹదͨ͠ܗʹฒସ͑ͳͲ 3".ʹಡΈॻ͖σʔλΛΩϟογϡ ݸʑͷϋʔυΣΞʹ͋Θͤͨॲཧ ε τ Ϩ c δ σ ό Π ε ந Խ %FWJDF NBQQFS ϒϩοΫͷಡΈॻ͖ཁٻΛ ͍Ζ͍Ζม
Device mapper (1) • ෳͷཧϒϩοΫσόΠεΛҰͭͷཧσόΠεͱͯ͠ ଋͶΒΕΔ • Mirror, Stripe, Snapshot
• RAID 0,1,5,10 • Snapshot: ཧσόΠεͷશมߋΛཧσόΠεʹϦμ ΠϨΫτ(Copy on Write) • ͋ͱͰཧσόΠεͱཧσόΠεΛϚʔδՄೳ • ཧσόΠε͔ΒόοΫΞοϓσόΠεʹϦϞʔτϦμ ΠϨΫτͰόοΫΞοϓ࡞ΕΔʁʁ
Device mapper (2) • 2ͭͷࢹ͕͔͚͍ͯΔ -> Snapshot Ͱແཧ • શมߋΛҰ࣌తʹཧσόΠε͕όοϑΝ͢Δඞ
ཁ͕͋Δ όοϑΝ͕͋;Εͯσʔλϩετ • ΦϦδφϧσʔλΛόοΫΞοϓઌͰϚʔδ͢Δ ඞཁ͕͋Δ • Device mapper ϒϩοΫมߋΛτϥοΩϯά͢Δ ͨΊͷશͯͷใΛͭ • liner mapping mode ࣌ͷ’map’ function
Architecture • ཧσόΠεʹରͯ͠τϥοΩϯά͢Δ͔ΘΓʹɺɹ ϧʔϓόοΫσόΠεʹରͯ͠τϥοΩϯά͢Δ • ϧʔϓόοΫσόΠε: ҰൠతͳϑΝΠϧΛϒϩοΫ σόΠεͰ͋Δ͔ͷΑ͏ʹѻ͏ͨΊͷػೳ ϒϩοΫσόΠε %FWJDFNBQQFS
ΞϓϦέʔγϣϯ ϒϩοΫσόΠε ϑΝΠϧγεςϜ ϧʔϓόοΫσόΠε %FWJDFNBQQFS ΞϓϦέʔγϣϯ
Data Structure • RAM্ʹมߋใΛ1ϒϩοΫ͋ͨΓ1bitͰͭ • 1bit ͷཁૉΛͭϒϩοΫͷྻ • ϝϞϦΞϩέʔγϣϯͷ •
kmalloc(), __get_free_pages(), vmalloc() • vmalloc() ͷΈ࣮֬ʹϝΨόΠτ୯ҐͰ֬อՄೳ • kmalloc() εϥϒΦϒδΣΫτ੍ݶ͕͋Δ(32MiB)ɺvmalloc ϖʔδ୯ҐͰ֬อ • ΦϯϝϞϦͳσʔλߏͳͷͰɺγϟοτμϯ࣌ʹτϥοΩϯ άใΛϩετ͢Δ • γϟοτμϯ࣌ʹτϥοΩϯάใΛσΟεΫʹॻ͖ࠐΜͰɺ ىಈ࣌ʹಡΈग़͢ • յΕ͍ͯΕϑϧಉظ
Evaluation (tools) • scp/nc • rsync • blockmd5sync • rsync
ͷϩʔϦϯάνΣοΫαϜͳ͠൛ • ZFS • features: ཧϘϦϡʔϜɺsnapshotɺ2ͭͷsnapshotͷࠩநग़ • ϒϩοΫσόΠεϨϕϧͩͱΞΫηεͰ͖ͳ͍ใ: /tmp ͚ͩແࢹ͢Δ • dsync • ϑΝΠϧγεςϜʹґଘ͠ͳ͍ • ϑΝΠϧγεςϜͷใ͕ͳ͍ͷͰ੍ݶ͋Δ (mtime ͳͲ) • νΣοΫαϜܭࢉͷ͔ΘΓʹɺϒϩοΫ͝ͱʹτϥοΩϯάεςʔλεΛͨ ͤΔ
Evaluation (Benchmarks) • 6-core AMD Phenom II processor • 2
TB spinning disk (Samsung HD204UI), • 128 GB SSD (Intel SSDSC2CT12) • εΠον͝͠ʹΪΨϏοτΠʔαωοτͰଓ
ಉظ࣌ؒ HDD/SSD rsync: 33min, dsync: 7 min )%% 44%
CPUར༻ STZODड৴ଆνΣοΫαϜ STZODૹ৴ଆνΣοΫαϜ ίΞ͍ͬͯΔ
ωοτϫʔΫଳҬ
Discussion • ৗʹ dsync > rsync • rsync dsync
ͷεʔύʔηοτ • rsync dsync ͱಉ͡Α͏ʹશͯͷߋ৽͞ΕͨϒϩοΫ Λread/transmit/merge͢Δ • rsync dsync ʹՃ͑ͯɺߋ৽ϒϩοΫΛܾఆ͢ΔͨΊ ʹɺ”શͯͷϒϩοΫ”Λreadɺchecksumܭࢉ͕ඞཁ • bit vector ͷߋ৽Φʔόϔου • ΦϯϝϞϦ͔ͩΒେͨ͜͠ͱͳ͍
Conclusion • ఆظతͳڊେόΠφϦσʔλͷޮతͳಉظํ ๏ͷఏҊ/࣮ • νΣοΫαϜΛܭࢉ͢ΔΘΓʹɺΦϯϥΠϯ ͰมߋΛτϥοΩϯά • Linux kernel
ͷ Device mapper֦ு • dmextract and dmmerge • rsycnc vs dsync, 32 min vs 7 min
ײ • rsync ͕͕͢͞ʹ͔Θ͍ͦ͏ͳͷͰɺసૹྔ͕͖͍ͯ ͘ΔΠϯλʔωοτܦ༝ͷಉظ࣌ؒൺֱ͕͋ΔͱΑ͞ ͦ͏
Linux 3.2 kernel module patch https://bitbucket.org/tknauth/devicemapper/