Hadoop初心者が脱初心者したかった話

by onomotoharu

Slide 1

Slide 1 text

Hadoop ॳ৺ऀ͕୤ॳ৺ऀ͔ͨͬͨ͠࿩ SkillWednesday#20161221 খ໺ݩय़

Slide 2

Slide 2 text

୭ খ໺ݩय़ BLBೋ࿠ *%நग़ج൫νʔϜ ΢ΠεΩʔɹ Ṍ͚ࣄ ϚΠϯΫϥϑτ

Slide 3

Slide 3 text

ͳͥ)BEPPQ

Slide 4

Slide 4 text

طଘΫϥελ͕Ϡό͍ Ҡߦ͠ͳ͖Ό

Slide 5

Slide 5 text

ٕज़໨ඪʹྑͦ͞͏ શવ͠Βͳ͍ྖҬͩ͠

Slide 6

Slide 6 text

ࠓ೔͸)BEPPQͷ࿩͸͠·ͤΜ

Slide 7

Slide 7 text

΍ͬͨ͜ͱ ࣌ظॱ

Slide 8

Slide 8 text

ωοτهࣄΛಡΉ ຊΛಡΉ ͱΓ͋͑ͣ࡞ͬͯΈΔ ಈ͍͍ͯΔ΋ͷΛ৮Δ ϕςϥϯʹฉ͘

Slide 9

Slide 9 text

ωοτهࣄΛಡΉ ˒ˑˑ ຊΛಡΉ ˒ˑˑ ͱΓ͋͑ͣ࡞ͬͯΈΔ ˑˑˑ ಈ͍͍ͯΔ΋ͷΛ৮Δ ˒˒ˑ ϕςϥϯʹฉ͘ ˒˒˒ Φεεϝ౓

Slide 10

Slide 10 text

ωοτهࣄΛಡΉ ˒ˑˑ

Slide 11

Slide 11 text

ಛʹࢀߟʹͳͬͨهࣄ • YARNͷ঺հ • https://www.ibm.com/developerworks/jp/ analytics/library/bd-yarn-intro/ • Hadoop͸ͲͷΑ͏ʹಈ͘ͷ͔ • http://gihyo.jp/admin/serial/01/ how_hadoop_works

Slide 12

Slide 12 text

Ͳ͏ͩͬͨͷ • ·ͣ͸“Hadoop ೖ໳”ͳͲͰάάͬͯग़ͯདྷΔ SlideshareΛಡΜͰΈΔ • ຊ౰ʹ;ΜΘΓͱ͔͠Θ͔Βͳ͍ • ͻͱ·ͱ·Γͷࢿྉ͸͍͖ͳΓಡΉͷ͸ϋʔυϧ ͕ߴ͍ • ܲͱཛঢ়ଶ • ඇৗʹ͕͔͔࣌ؒΔ • ࠓͷཧղ౓ʹదͨ͠ࢿྉ͔͸ࣗ෼Ͱ൑அෆՄ

Slide 13

Slide 13 text

ຊΛಡΉ ˒ˑˑ

Slide 14

Slide 14 text

ಡΜͩຊ • HadoopΦϖϨʔγϣϯ  -γεςϜӡ༻؅ཧΨΠυ

Slide 15

Slide 15 text

Ͳ͏ͩͬͨͷ • Ͳ͔͜ΒಡΊ͹͍͍ͷ… • ۀ຿ͳͲͰ৮Εͨ͜ͱͱ཭Ε͍͗ͯ͢Δ • “Ͳ͏خ͍͠”ͷ͔Πϝʔδ͕෇͔ͳ͍ • ਖ਼௚͋·Γ಺༰͍֮͑ͯͳ͍Ͱ͢  …ͷͰ͜ͷ̍࡭͔͠ಡΊͳ͔ͬͨ

Slide 16

Slide 16 text

ͱΓ͋͑ͣ΍ͬͯΈΔ ˑˑˑ

Slide 17

Slide 17 text

΍ͬͨ͜ͱ • QuickStarts for CDH 5.8 • shྲྀ͢ͱΫϥελ͕Ͱ͖Δ • ͍͖ͳΓCMυϯͬͯͳΔ • http://www.cloudera.com/ downloads/ quickstart_vms/5-8.html

Slide 18

Slide 18 text

Ͳ͏ͩͬͨͷ • Ϋϥελ͸ཱͬͨʂద౰ͳCSVΛೖΕͯHive Ͱୟ͍ͯΈͨʂ • …͔͜͜ΒԿΛ͢Ε͹͍͍ͷ͔ʁ • ͦ΋ͦ΋ଞσΟετϦϏϡʔγϣϯͳͲͱൺ ֱ͢ΔͨΊͷ෺ͬΆ͍ • ࣗ෼Ͱࢼߦࡨޡ͢Δ෦෼͕ͳ͍

Slide 19

Slide 19 text

ಈ͍͍ͯΔ΋ͷΛ৮Δ ˒˒ˑ

Slide 20

Slide 20 text

΍ͬͨ͜ͱ • CosmoͷEMRͷઃఆΛ(ͨͿ Μ)શ෦ಡΜͩ • ࣗ෼ͳΓʹཧղ্ͨ͠Ͱಉ͡ ઃఆͰΫϥελ࡞੒ • …ͷલʹAWSͷࢿྉ΋ಡΈ ͭͭϓϨʔϯͳ΋ͷΛ࡞੒

Slide 21

Slide 21 text

Ͳ͏ͩͬͨͷ • AWSͷࢿྉΛಡΈͳ͕ΒཱͯͯΈΔ • EMRͷػೳ΍ࢥ૝͕;ΜΘΓ෼͔Δ • طଘͷઃఆΛಡΈͳ͕ΒཱͯΔ • ͳͥ͜ͷઃఆ͕͍Δͷ͔ɺάάΔɺҙਤ͕ ෼͔ΔˠεοΩϦ&๨Εͳ͍ • ࣗ෼ͷPJͰͷԠ༻͸·ͩԕͦ͏

Slide 22

Slide 22 text

ϕςϥϯʹฉ͘ ˒˒˒

Slide 23

Slide 23 text

΍ͬͨ͜ͱ • ಺༰ • ࣮ࡍʹ࡞ۀ͍ͯ͠Δͱ͖ʹෆ໌఺ • ೲಘ͍͔ͳ͍ࣄ • ۩ମతͳϢʔεέʔε͕ු͔͹ͳ͍ύλʔϯ • ํ๏ • slackͰ෦԰ཱͯ(#hadoop) • ͦͷ৔Ͱ࢕͏

Slide 24

Slide 24 text

Ͳ͏ͩͬͨͷ • “શମײ͔ͭΊͨͭ΋Γ…͚ͩͲ͕݀͋Δ”ͷ ݀Λ࠹͛Δ • ࣗ෼ͷཧղ౓/PJͷཁ͕݅͋Δఔ౓ڞ༗ࡁΈ  ཧղ౓ʹ߹Θͤͨձ࿩(ΠϯλϥΫςΟϒ!)

Slide 25

Slide 25 text

ͱ͸ݴ͍ͭͭ΋ ·ͱΊͱײ૝

Slide 26

Slide 26 text