spark shuffle 勉強会

spark shuffle 勉強会

Transcript

1. SparkͷShuffleपΓ @huydx
2. Shuffleʹ͍ͭͯ • Map - Reduce Ϟσϧ • Mapஈ֊͔ΒReduceஈ֊΁ͷதؒϨΠϠʔ͸ ʮShuffleʯͱݺͿ • ShuffleͰ͸ • Spark͕PullϞσϧʢ·ͣσΟεΫʹ݁Ռॻ͍ ͯɺReduceδϣϒ͕औΓʹߦ͘ʣ • Spark͸Reduceδϣϒʹඞཁͳσʔλ͸ϝϞϦ ϑΟοτ͠ͳ͍ͱ͍͚ͳ͍
3. ͍ͭShuffle͕ൃੜ͢Δ • Join • Cogroup • *ByKeyΦϖϨʔγϣϯ
4. Shuffleͷ໰୊ • ShuffleϑΝΠϧ਺ • Mapͷ਺͕MɺReduceͷ਺͕Rͱͨ͠ΒσΟεΫʹॻ͘ ϑΝΠϧ਺͕ M * R (M = 5000, R = 1024 ͩͱ 500ສϑΝ Πϧʂʣ • Reduce͢Δͱ͖ʹιʔτΞϧΰϦζϜ͕ඞཁ • ฒྻʹιʔτ͢Δඞཁ͕Ͱ͖Δ΋ͷ • ௨৴͕ॏ͍
5. Shuffleͷ໰୊ղܾ • ShuffleϑΝΠϧ਺ɿ • O(M * R) ͡Όͳͯ͘ O(R)·Ͱ཈͑ΒΕΔ • Hashed base shuffle(ҰͭͷRͻͱͭͷϑΝΠϧʣ͡Όͳͯ͘ Sort base shuffle • ࢀߟɿhttps://issues.apache.org/jira/secure/attachment/ 12637642/Consolidating%20Shuffle%20Files%20in %20Spark.pdf • https://issues.apache.org/jira/browse/SPARK-2045
6. Shuffleͷ໰୊ղܾ • SortͷΞϧΰϦζϜબ୒ • https://databricks.com/blog/2014/10/10/spark- petabyte-sort.html • TimsortΛ࣮૷͢Δ • ৭ʑͳιʔτΞϧΰϦζϜͷ૊Ή߹ΘͤͰฏ ۉWorst CaseύʔϑΥϚϯεΛݮΒ͢
7. Shuffleͷ໰୊ղܾ • ωοτϫʔΫϞδϡʔϧΛվળ • https://issues.apache.org/jira/browse/ SPARK-2468 • Netty ϕʔεσʔλసૹͷ࣮૷ (FileChannel.transferToͰzero copyʣ