$30 off During Our Annual Pro Sale. View Details »

DatabricksとSparkではじめる [データ分析/機械学習] 実践入門 / Databrick and Spark with Data Analyze and ML for newbie.

DatabricksとSparkではじめる [データ分析/機械学習] 実践入門 / Databrick and Spark with Data Analyze and ML for newbie.

Scala関西Summit 2018

Masatoshi Shimada

November 09, 2018
Tweet

More Decks by Masatoshi Shimada

Other Decks in Technology

Transcript

  1. %BUBCSJDLTͱ4QBSLͰ͸͡ΊΔ
    <σʔλ෼ੳػցֶश>
    ࣮ફೖ໳
    ౡాխ೥
    4DBMBؔ੢4VNNJU

    View Slide

  2. ࣗݾ঺հ
    w ౡాխ೥!TNENUT
    w σʔλ΢ΣΞϋ΢εΞʔΩςΫτ
    4DBMBc1ZUIPOc(PMBOHc+BWBc%%%

    w גࣜձࣾΤϒϦʔॴଐ
    w %&-*4),*5$)&/σʔλ෼ੳܥશൠ୲౰
    ओʹ෼ੳج൫ɺ഑৴໘࠷దԽͳͲΛ୲౰

    View Slide

  3. ΞϓϦϦϦʔε
    μ΢ϯϩʔυɿສ
    Ϩγϐ਺ɿສ݅Ҏ্
    ݄ؒಈը࠶ੜ਺ɿԯສճҎ্
    "QQ4UPSF#&450'
    (PPHMF1MBZϕετΦϒ
    ೥ࠃ಺ΞϓϦμ΢ϯϩʔυୈ̏Ґ
    ʢ̍Ґ-*/&ɺ̎Ґ:BIPP+"1"/˞ඇήʔϜɹ"QQ"OOJFௐ΂ʣ
    ɾຖ೔৽ணϨγϐಈը͕഑৴͞ΕΔ
    ɹɾྉཧϨγϐಈըʢௐཧݙཱखॱʣ
    ɹɾ޿ࠂओλΠΞοϓಈը
    <13>%&-*4),*5$)&/ͷ঺հʢ͔ΜͨΜʹʣ

    View Slide

  4. ΞδΣϯμʢ֓ཁʣ
    %&-*4),*5$)&/Ͱ͸ɺσʔλ෼ੳͷجװٕज़ͷҰͭͱͯ͠
    4QBSLͷϗεςΟϯάαʔϏεͰ͋Δ%BUBCSJDLTΛར༻͍ͯ͠·͢
    ຊ೔͸αϯϓϧίʔυΛަ͑ͯɺ%BUBCSJDLTͷ֓ཁͱ
    σʔλ෼ੳػցֶशͰར༻͍ͯ͠Δ4QBSLͷίʔυʹ͍ͭͯ
    ղઆͤͯ͞௖͖·͢

    View Slide

  5. ΞδΣϯμ
    w %BUBCSJDLTͷ֓ཁɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ
    ίϯηϓτΠϯϑϥߏ੒ྫ/PUFCPPL࣮ߦσϞ
    w %BUBCSJDLTʢ4QBSLʣͷίʔυ࣮૷ྫɹɹɹɹɹɹɹɹɹɹɹ
    ؆୯ͳαϯϓϧίʔυػցֶशʢڠௐϑΟϧλϦϯάʣɹɹɹ
    ෇࿥ɿσʔλ෼ੳʢϚδοΫφϯόʔ෼ੳʣ

    View Slide

  6. %BUBCSJDLTͷ֓ཁ
    ίϯηϓτΠϝʔδ
    Πϯϑϥߏ੒ྫ
    /PUFCPPL࣮ߦσϞ

    View Slide

  7. %BUBCSJDLTͷίϯηϓτΠϝʔδ

    View Slide

  8. %BUBCSJDLTͷίϯηϓτΠϝʔδ
    ɾ4QBSLΛ࣮ߦج൫ͱͨ͠σʔλ෼ੳϓϥοτϑΥʔϜ
    6OJpFE"OBMZUJDT1MBUGPSNΛ໨ࢦ͍ͯͯ͠࡞ΒΕ͍ͯΔ
    ɾ "84"[VSF
    ͷϢʔβΞΧ΢ϯτʹ71$࡞੒ͳͲॴఆͷΠϯετʔϧͰಋೖ
    ར༻ऀͷ "84"[VSF
    ΞΧ΢ϯτͰΫϥελ͕ىಈ͞ΕΔ
    ɾଞͷػցֶशج൫ͱͷͭͳ͗ࠐΈ͕Մೳ
    5FOTPS'MPX9(#PPTUTDJLJUMFBSO౳
    ɾσʔλΤϯδχΞαΠΤϯςΟετ͸
    ɹ/PUFCPPLͰεΫϦϓτΛॻ͍ͯ։ൃ͢Δ
    4DBMB1ZUIPO34IFMM42-͕ར༻Մ
    &5-ॲཧ΋/PUFCPPLͰ࣮૷Մ

    View Slide

  9. Πϯϑϥߏ੒ྫʢ%&-*4),*5$)&/ͷ৔߹ʣ
    %BUBCSJDLT71$
    ΞϓϦέʔγϣϯ71$
    εέʔϥϒϧ
    71/PS*OUFSOFU
    71$ϐΞϦϯά઀ଓ
    4
    &MBTUJ
    $BDIF
    3%4 3FETIJGU
    %BUBCSJDLT
    $MPVE
    ʢ؅ཧը໘ʣ
    ىಈઃఆ
    /PUFCPPL

    View Slide

  10. %BUBCSJDLTʹΑΔܧଓతσϦόϦ
    ɾ%BUBCSJDLT$MPVEͷ/PUFCPPL্Ͱσʔλ෼ੳίʔυͷ։ൃΛߦ͏
    ɾ$-*ʹΑΔσϓϩΠ΍Ϋϥελ࡞੒౳͕ग़དྷͯ$*$%Ͱ΋ར༻Մೳ
    ɾैདྷͷTQBSLTVCNJUͰɺKBS͔ΒNBJOΫϥεΛىಈ͢Δํࣜ΋ར༻Մೳ
    1VMM
    1VTIBOE3VO
    1VMM
    1VTI %FQMPZ
    PS
    .FSHF
    ຊ൪؀ڥ
    ։ൃ؀ڥ
    ʢ։ൃऀͷϫʔΫεϖʔεʣ
    ຊ൪؀ڥ

    View Slide

  11. %BUBCSJDLT؅ཧը໘ʢτοϓʣ
    ɾ։ൃऀʢσʔλΤϯδχΞσʔλαΠΤϯςΟετʣ͸
    ɹ%BUBCSJDLT$MPVEͷ63-Λ։͖ɺ؅ཧը໘ʹ઀ଓͯ͠։ൃΛ࢝ΊΔ

    View Slide

  12. %BUBCSJDLT؅ཧը໘ʢΫϥελઃఆʣ
    ɾ4QBSLΫϥελͷΠϯελϯελΠϓઃఆը໘
    ɹΦʔτεέʔϧىಈΫϥελ $16(16
    4QBSLόʔδϣϯ౳͕ࢦఆՄೳ
    εϙοτೖࡳ
    ΠϯελϯελΠϓ
    ୆਺ׂ߹
    Φʔτεέʔϧઃఆ
    Φʔτλʔϛωʔτઃఆ

    View Slide

  13. %BUBCSJDLT؅ཧը໘ʢ/PUFCPPLʣ
    ɾ։ൃऀຖʹׂΓ౰ͯΒΕͨ8PSLTQBDF্ͷɺ/PUFCPPLͰ։ൃΛ͢͢ΊΔ
    ɹ4DBMB1ZUIPO342-͕બ୒ՄೳʢͲΕΛબΜͰ΋૬ޓݺͼग़͠Ͱ͖Δʣ

    View Slide

  14. ɾ/PUFCPPL্ʹෳ਺ͷίϚϯυΛ࣮૷͠ɺ4UFQCZ4UFQͰॲཧΛ࣮ߦ͢Δ
    ΠϯλϥΫςΟϒʢର࿩తʣʹίϚϯυ͕ஞ࣮࣍ߦ͞ΕΔ
    ɾίϚϯυؒͷσʔλ࿈ܞ͸ɺ4DBMBͷΠϯελϯε΍4QBSL42-Ͱߦ͏
    ɾ42-%4-ͳͲͰ్தஅ໘ͷσʔλΛ֬ೝ͠ͳ͕Β։ൃΛ͢͢ΊΔ
    /PUFCPPLʹΑΔ։ൃͷ͢͢Ίํ
    5%4QBSL

    View Slide

  15. /PUFCPPL࣮ߦσϞ

    View Slide

  16. %BUBCSJDLTʢ4QBSLʣͷίʔυ࣮૷ྫ
    ؆୯ͳαϯϓϧίʔυ
    ػցֶश ڠௐϑΟϧλϦϯά

    ෇࿥ɿσʔλ෼ੳ ϚδοΫφϯόʔ෼ੳ

    View Slide

  17. IUUQTHJUIVCDPNTNENUTECDTDBMB@LT@
    αϯϓϧίʔυ͸044ͱͯ͠ެ։͍ͯ͠·͢
    ͪ͜ΒΛ͝ࢀর͍ͩ͘͞

    View Slide

  18. ؆୯ͳαϯϓϧίʔυ

    View Slide

  19. σʔλఆٛ
    7*&8࡞੒
    42-࣮ߦ
    σʔλఆٛ
    7*&8࡞੒

    View Slide

  20. 42-࣮ߦ
    4DBMBίʔυ

    View Slide

  21. ݁Ռ֬ೝ
    1ZUIPOίʔυ

    View Slide

  22. ػցֶश
    ڠௐϑΟϧλϦϯά

    View Slide

  23. ڠௐϑΟϧλϦϯάͱ͸
    ୭͔͕ݟͨϨγϐΛجʹɺର৅ऀʹ͓͢͢Ί͢ΔϨγϐΛࢉग़͢Δݹయతػցֶश
    ࣅͨΑ͏ͳϨγϐΛݟͨϢʔβಉ࢜Ͱɺ͓ޓ͍ʹະࢹௌͷϨγϐΛަ׵͢Δ
    ʢυϝΠϯ஌͕ࣝແͯ͘΋ҎԼͷΑ͏ʹࢉग़Մೳʣ





    ͕ࢹௌࡁΈͷϨγϐ

    ʹఏҊ͢ΔະࢹௌͷϨγϐ




    View Slide

  24. σʔλఆٛ
    ʢ௨ৗ͸%#΍4ͳͲʣ

    View Slide

  25. 7JFX࡞੒
    *OEFYFS࡞੒
    ܭࢉྔ࡟ݮͷͨΊ*OUԽ͢Δఆٛ
    ػցֶश
    ܭࢉࣜఆٛ

    View Slide

  26. ܭࢉ༻
    ύΠϓϥΠϯ࡞੒
    7JFX࡞੒
    σʔλՄࢹԽ

    View Slide

  27. *OEFYFS݁Ռఆٛ
    *OUVTFS@JEͷඥ෇͚༻7*&8
    *OEFYFS݁Ռఆٛ
    Ϩγϐ໊VTFS@JEͷ
    ඥ෇͚༻7*&8

    View Slide

  28. VTFS@JE
    JOEFYࡁΈVTFS@JE
    Ϩγϐ໊
    JOEFYࡁΈϨγϐ໊
    *OEFYFS݁Ռ֬ೝ
    *OEFYFS݁Ռ֬ೝ

    View Slide

  29. σʔλՄࢹԽ
    Ϟσϧܭࢉ
    Ϩίϝϯυࢉग़
    7JFX࡞੒

    View Slide

  30. ݁Ռऔಘ7*&8

    View Slide

  31. ݁Ռ֬ೝ

    View Slide

  32. ڠௐϑΟϧλϦϯά݁Ռ


    ͕ࢹௌࡁΈͷϨγϐ


    ʹఏҊ͢ΔະࢹௌͷϨγϐ





    ΦϨϯδͷํ͕൓Ԡ͕ڧ͍





    View Slide

  33. ࣭ٙɾԠ౴

    View Slide

  34. ෇࿥

    View Slide

  35. σʔλ෼ੳ
    ϚδοΫφϯόʔ෼ੳ

    View Slide

  36. ϚδοΫφϯόʔ෼ੳͱ͸
    ྫ͑͹ɺ͜ͷαϜωΠϧ෦෼ΛΫϦοΫͨ͠ࣄͷ͋ΔϢʔβͷܧଓ཰͸ɺͲͷ͘Β͍ͩΖ͏͔ʁ
    ʮ͓ؾʹೖΓͨ͠ಈըࢹௌͨ͠ݕࡧͨ͠ʯ౳ɺผͷΞΫγϣϯͱൺ΂ܧଓ཰͸ߴ͍ͩΖ͏͔ʁ
    ܧଓ཰͕ߴ͍ͷ͸ͲΜͳϢʔβͩΖ͏͔ʁ؀ڥ΍ଐੑ͸ʁ
    ϚδοΫφϯόʔ෼ੳͱ͸ɺ༷ʑͳཁૉΛ૊Έ߹Θͤͨ৚݅ ूஂ
    ͷத͔Β
    ܧଓ཰ͷߴ͍ϢʔβΞΫγϣϯͷ܏޲Λൃݟ͍ͯ͘͠෼ੳख๏ͷ͜ͱɻ
    ྑ͍ΞΫγϣϯΛݟ͚ͭͨΒɺͦͷಋઢΛڧԽ͍͚ͯ͠͹ྑ͍ɻ
    ʮର৅ΞΫγϣϯɾϢʔβଐੑɾܧଓ೔਺ʯΛ૊Έ߹Θͤɺ֤ूஂͷܧଓ཰Λಋ͖ൺֱ͢Δɻ
    ֤ूஂͷΞΫγϣϯຖͷܧଓ཰Λࢉग़͢Δ
    ܧଓ཰ಉ࢜Λൺֱ͠෼ੳ͢Δ
    ॲཧϑϩʔ

    View Slide

  37. ֤ूஂͷΞΫγϣϯຖͷܧଓ཰Λࢉग़͢Δ

    ʢॳ೔ʣ
    ʢ̎೔໨ʣ
    ʢ೔໨ʣ









    ࢉग़ํ๏



    ྫɿαϜωΠϧΛݟͨਓ

    View Slide

  38. σʔλఆٛ
    ʢ௨ৗ͸%#΍4ͳͲʣ

    View Slide

  39. 7JFX࡞੒
    σʔλՄࢹԽ

    View Slide

  40. ܧଓ཰ܭࢉ

    View Slide

  41. ܧଓ཰ܭࢉ
    ݁ՌՄࢹԽ

    View Slide

  42. ݁ՌՄࢹԽ

    View Slide


  43. ʢॳ೔ʣ
    ʢ̎೔໨ʣ
    ʢ೔໨ʣ









    ص্ܭࢉͷ݁ՌͱҰக



    ֤ूஂͷΞΫγϣϯຖͷܧଓ཰ࢉग़݁Ռ

    View Slide

  44. ܧଓ཰ಉ࢜Λൺֱ͠෼ੳ͢Δ
    wαϜωΠϧΛݟͨूஂ ೔
    ˋ
    w͓ؾʹೖΓΛͨ͠ूஂ ೔
    ˋ
    ࢉग़ͨ݁͠Ռ্͕هͷΑ͏ʹͳͬͨ৔߹
    ͓ؾʹೖΓ΁ͷಋઢΛڧԽ͢Ε͹ྑ͍ͱ෼͔Δ

    View Slide

  45. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide