PyCon JP 2017 の発表資料です。ジャンルは「業務利用事例」となります。
追記1. PyCon JP 2017 ベストトークアワード優秀賞を受賞しました! 追記2. http://yuzutas0.hatenablog.com/entry/2017/09/12/203000 に補足を掲載しています!
Jupyter + BigQuery ʹΑΔ σʔλੳج൫ ͷ DevˍOps2017-09-09 (Sat) 4:15 p.m. – 4:45 p.m. in Room 201 PyCon JP 2017 presented by @yuzutas0 https://www.pexels.com/photo/close-up-of-computer-keyboard-257949/ɹhttps://www.pexels.com/photo/technology-computer-lines-board-50711/ɹhttps://www.pexels.com/photo/black-and-white-business-chart-computer-241544/
View Slide
ຊͷࢿྉWEBʹެ։͠·͢ #pyconjp #pyconjp_201 ࡱӨɾهͷඞཁ͋Γ·ͤΜ ͭ͘Ζ͍Ͱௌ͍͍͚ͯͨͩΕͱࢥ͍·͢
εϥΠυ 150+ / 30min ʢݟΛ͕ͬͭΓڞ༗͠·͢ʣ ͕Μ͕Μߦ͖·͢ʂ
ɹSho Yokoyamaɹ@yuzutas0ɹɹ ● Recruit Technologies Co., Ltd. ɹɹάϧʔϓͷITࢪࡦશൠΛ୲͏ byʰϦΫϧʔτɺਐԽΛࢭΊͳ͍ITݱྗʱ● Site Reliability Engineering ɹɹج൫γεςϜͷӡ༻ɾվળ● Finance & Econometrics (Before) ɹɹRʹΑΔ౷ܭॲཧ - ͋ͷͱ͖Jupyter͕͋ΕΑ͔ͬͨͷʹʂ
https://github.com/yuzutas0ੳج൫ɹޱखಈ͔͢
ϓϩμΫτ։ൃ ʹ͓͚Δ σʔλ׆༻ ͷ ࢀߟࣄྫ ͷڞ༗ɹຊͷΰʔϧ
Python Λ׆༻ͯ͠ ։ൃݱ ʹ σʔλจԽ Λਁಁ͍ͤͨ͞ ιϑτΣΞΤϯδχΞɹఆ͢Δର
ɹɹɹɹPythonͷɹɹɹ"ɹPythonΛۀͰ͏ɹPythonͷConferenceͰ͕͢த൫·Ͱग़͖ͯ·ͤΜʢʣ
1. ϓϩμΫτ / νʔϜ 2. ੳج൫ͷDev&OpsɹɹϢʔεέʔεɹߏஙͷഎܠɹઃܭϙϦγʔɹσʔλϑϩʔɹ։ൃϓϩηε 3. σʔλจԽΛ৫ʹண͢ΔɹΞδΣϯμ
ࠗ׆ ࿀׆
ֹ݄՝ۚϞσϧ ※࿀݁ͼঁੑϢʔβʔຊਓ֬ೝྉͷΈɹ͝ར༻ͷྲྀΕ
※αϒγεςϜͷٕज़ཁૉ: Golang, Groovy, Python, Ruby, AWS, GCPɹɹɹɹɹɹɹɹɹɹɹɹΦϯϓϨɹ֤छσόΠεରԠ - γεςϜུ֓ (AP)
※ถࠃͰ݁ࠗͨ͠Χοϓϧͷ1/3͕ΦϯϥΠϯͰͷग़ձ͍ɹσʔςΟϯάࢢͷ֦େ
ɹࣄۀ → εςʔΫϗϧμʔͷଟ༷ԽσβΠϯηΩϡϦςΟ๏ΠϯϑϥSREΞϓϦج൫ू٬ΧελϚʔ αϙʔτσʔλ αΠΤϯεػցֶश ΤϯδχΞ։ൃӡ༻ νʔϜA։ൃӡ༻ νʔϜB։ൃӡ༻ νʔϜCPO PO POσΟϨΫγϣϯใ
ɹ֤෦ॺͷσʔλఏڙσʔλج൫
ɹͲͷΑ͏ʹ׆༻͍ͯ͠Δ͔σʔλج൫
ɹσʔλ׆༻ࣄྫɹ1. KPIϞχλϦϯάɹɹ ɹ2. ϢʔβʔߦಈͷՄࢹԽ ɹ3. ࢪࡦޮՌͷ༧ଌɾܭଌϏδωε4. ػցֶशʹΑΔϨίϝϯυ 5. ۀऀɾεύϜͷࣗಈఆϓϩμΫτɹ6. ͍߹Θͤͷੳ ɹ7. ࠂ৴ͷࣗಈ࠷దԽ ɹ8. ϓϨεϦϦʔεΧελϚʔνϟωϧɹ9. γεςϜϞχλϦϯά ɹ10. োൃੜ࣌ͷӨڹௐࠪγεςϜɹ11. νʔϜͷΩϟύγςΟνʔϜ
Ϗδωεࢦඪʢച্DAUʣͷਪҠ ຖேͷSlack௨ɹɹɹɹSpreadsheetɹɹɹɹμογϡϘʔυɹɹɹKPIϞχλϦϯά1
ओཁಋઢʢձһొˠϝοηʔδʣͷར༻ɾߦಈϑΝϯωϧɹɹɹɹɹɹδϟʔχʔϚοϓɹɹɹϢʔβʔߦಈͷՄࢹԽ2
Split (AB) ςετ݁Ռɹɹ৽ػೳͷར༻߹͍ɹɹɹࢪࡦͷޮՌ༧ଌˍܭଌ3
ɹɹɹػցֶशʹΑΔϨίϝϯυ4
ɹɹɹۀऀɾεύϜͷࣗಈఆ5
ɹɹɹ͍߹Θͤͷੳ6
ɹɹɹࠂ৴ͷࣗಈ࠷దԽ7
σʔλੳ݁Ռͷެ։ → ࢢͷ׆ੑԽʹߩݙɹɹɹϓϨεϦϦʔε8[email protected]
ΫϥογϡϨεϙϯελΠϜɹɹɹγεςϜϞχλϦϯά9IUUQZV[VUBTIBUFOBCMPHDPNFOUSZ
ɹɹɹোൃੜ࣌ͷӨڹௐࠪ10
εΫϥϜνʔϜͷϕϩγςΟ νέοτফԽͷϝτϦΫεɹɹɹνʔϜͷΩϟύγςΟ11
σβΠϯηΩϡϦςΟ๏ΠϯϑϥSREΞϓϦج൫ू٬ΧελϚʔ αϙʔτσʔλ αΠΤϯεػցֶश ΤϯδχΞ։ൃӡ༻ νʔϜA։ൃӡ༻ νʔϜB։ൃӡ༻ νʔϜCPO PO POσΟϨΫγϣϯใ։ൃӡ༻ νʔϜBPOɹ͜ΕΒͷࢪࡦ : ֤෦ॺ͕ϘτϜΞοϓͰਪਐ
σʔλ׆༻Ͱ࣮ݱͰ͖Δ͜ͱɹސ٬Ձͷٻ → σʔλ׆༻ඞવ1. ࣗಈԽ 2. ՄࢹԽސ٬Ձͷఏڙ
“ΑΓૣ͘ɾ҆ఆͨ͠” αʔϏεͷఏڙɹ1. ࣗಈԽػցֶशͰީิऀϦετΛࣄલநग़ࢹ͚ͩͰۀऀɾεύϜΛఆ[email protected] [email protected]
“ΑΓޮՌతͳ” ΤϯδχΞϦϯάͷ࣮ݱɹ2. ՄࢹԽ͜ͷσβΠϯϢʔβʔͷ ຊԻʹ͍͑ͯΔͷ͔ʁ͜ͷػೳՃޮՌ͕͋Δͷ͔ʁ ࣮ࡍʹޮՌ͋ͬͨͷ͔ʁIUUQHBUBHOFUGSFFJMMVTUSBJUJPO [email protected]
σβΠϯηΩϡϦςΟ๏ΠϯϑϥSREΞϓϦج൫ू٬ΧελϚʔ αϙʔτσʔλ αΠΤϯεػցֶश ΤϯδχΞ։ൃӡ༻ νʔϜA։ൃӡ༻ νʔϜB։ൃӡ༻ νʔϜCPO PO POσΟϨΫγϣϯใ։ൃӡ༻ νʔϜBPOɹ֤෦ॺʹΑΔੵۃతͳσʔλ׆༻
ɹͳͥج൫͕ඞཁʹͳ͔ͬͨσʔλج൫
DBσʔλΛ͍͍ͨͷͰ ࣍ͰS3ʹCSVग़ྗ͍ͯͩ͘͠͞ ɹ࠷ॳখ͞ͳσʔλૄ௨ґཔ෦ॺA
ɹεέʔϧ → ֤෦ॺ͔Βґཔ෦ॺB෦ॺC෦ॺD
ͬͪ͜Ͱαʔόͱૄ௨πʔϧΛ༻ҙ͠·͢ ϨϏϡʔˍຊ൪ڥʹஔ͍ͯΒ͑·͔͢ɹͪΐͬͱֻ͕͔࣌ؒΓͦ͏ੜσʔλΛͯ͠Β͑Ε ͬͪ͜Ͱ͑ΔΑ͏ʹՃ͢ΔͷͰ෦ॺE
ɹ֤෦ॺ͝ͱʹσʔλૄ௨෦ॺA πʔϧ෦ॺB πʔϧ෦ॺA αʔό෦ॺB αʔό
ɹ֤෦ॺ͝ͱʹσʔλՃ෦ॺA நग़σʔλ෦ॺB நग़σʔλ෦ॺA ՃϩδοΫ෦ॺB ՃϩδοΫ
ɹମ੍ͷ··ʹγεςϜෳࡶԽ - ίϯΣΠͷ๏ଇݩσʔλ֤෦ॺ͚ʹσʔλࢀরγεςϜΛܧ͗͠
http://jp.techcrunch.com/2017/06/25/20170623five-building-blocks-of-a-data-driven-culture/ελοϑ͕ͨͪҟͳΔγεςϜ͔ΒࣅͨΑ͏ͳϝτϦοΫΛ Ҿ͖ग़͢͜ͱ͕Ͱ͖ͯ͠·͏ͳΒɺඞવతʹͦΕΒͷγεςϜ͔Β ҟͳΔࣈ͕ੜ͞ΕΔɻ…ΑΓ·͠ͳใݯΛ͑Δഺͩͬͨͷʹɺ ݹ࣭ͯ͘ͷ͍ɺ͋Δ͍ؒҧͬͨσʔλࢦඪΛΒͣʹ ͬͯ͠·͍ɺѱ͍ܾఆΛԼ͢νʔϜग़ͯ͠·͏͔Εͳ͍ɻɹ෦ॺ͝ͱʹʮച্ʯ͕ζϨΔ
ɹΤϯδχΞʹௐࠪґཔᶃ - σʔλ༷ʑมΘΔ͜ͷϨίʔυ͕ Ճ͞ΕΔλΠϛϯάɺ ࠷ۙมߋ͞Ε·ͨ͠ʁͪΐͬͱมߋཤྺΛ ͬͯΈ·͢Ͷ෦ॺF ΤϯδχΞ
ɹΤϯδχΞʹௐࠪґཔᶄ - Өڹൣғʑ͕ΔRe:dashͷࣈ͕ ζϨΔΜͰ͚͢Ͳଞ෦ॺ͕࡞ͬͨγεςϜΛ ௐΔͱ͜Ζ͔Β……෦ॺG ΤϯδχΞ
ɹݸผରԠͷѱ॥ಠࣗσʔλૄ௨ͷґཔ ʢ͜ΕͳΒ͙͢Ͱ͖·͢ΑͶʣΤϯδχΞνʔϜͷෛՙˢγεςϜෳࡶԽˢௐࠪґཔ
ɹ୭͔ͷஅ͕ѱ͔ͬͨΘ͚Ͱͳ͍εέʔϧຊ֨Քಇ࣌ʹ ͕ݦࡏԽɹɹ·ͣখ͘͞ࢼ͢ ɹ ● ࠷খݶͷσʔλૄ௨ ɹ ● ࣗલͰσʔλՃ෦ॺ͝ͱʹ ৽πʔϧಋೖࢪࡦணख[email protected][email protected] [email protected]
http://jp.techcrunch.com/2017/06/25/20170623five-building-blocks-of-a-data-driven-culture/ɹମ੍֦େ → σʔλΣΞϋεͷඞཁੑࣄ࣮ͷ୯ҰใݯΛ͍࣋ͬͯΔ߹ʹɺΞφϦετ ଞͷҙࢥܾఆऀͱ͍ͬͨΤϯυϢʔβʔͨͪʹɺ༏ΕͨՁΛ ఏڙ͢Δ͜ͱ͕Ͱ͖Δɻ൴Β৫ͰσʔλΛ୳͕࣌ؒ͢ গͳͯ͘ࡁΉΑ͏ʹͳΓɺσʔλͷར༻ʹΑΓଟ͘ͷ࣌ؒΛ ׂ͘͜ͱ͕Ͱ͖ΔΑ͏ʹͳΔ͔Βͩɻ
ɹͲͷΑ͏ͳγεςϜΛߏங͔ͨ͠σʔλج൫
1. ModelͱViewΛ͚Δ͠ 2. ͳΔ͘ϥΫͯ͠࡞Δ͠ɹઃܭϙϦγʔ
ɹModelͱViewΛ͚Δ͠ɹɹɹɹɹɹɹɹɹɹɹɹɹDWHʹԠͨ͡γεςϜͷɾ݁߹ Model (ੵ/Ճ) ͱView (ࢀর)
ɹҙɿ෦ॺ͝ͱʹ࠷దͳViewҟͳΔExcelؾܰʹࣈΛม͑ͯ γϛϡϨʔγϣϯϏδωε෦TableauߴՁ֨ɾߴػೳ ੳཁٻʹରԠRe:dashSQL͕ॻ͚Δͻͱ ͓खܰར༻σΟϨΫλʔJupyter GitͰίʔυཧ ϓϩάϥϜͷԸܙΤϯδχΞੳ෦
ɹจ໌ͷརثσʔλૢ࡞ͳΒPythonҰΤίγεςϜʹΔσʔλอଘͳΒBigQueryҰ GoogleʹཔΔΞϓϦ Πϯϑϥ
ɹɹɹ● ʢRͱൺͯʣॻ͖͘͢ಡΈ࣮͘͢ߦ͍͢͠ ɹɹɹ● ൚༻తͳϓϩάϥϜॲཧΛॻ͚ΔεΫϦϓτݴޠɹPythonͷ࠾ཧ༝ - ൚༻LL
ɹɹɹɹ● ʢଞͷ൚༻ݴޠʹൺͯʣσʔλͷՃඳը͕ಘҙ ɹɹɹɹ● DB㱻DataframeมͳͲଞγεςϜ࿈ܞ༰қɹPythonͷ࠾ཧ༝ - σʔλૢ࡞͚ϥΠϒϥϦ
ɹɹɹ● σόοάσʔλͷՄࢹԽ͕ίϚϯυ1ͭͰՄೳ ɹɹɹ● ෦తʹjson → GitͰࠩཧ → ࠶ݱՄೳͳهΛͤΔɹPythonͷ࠾ཧ༝ - ࠷ڧͷ࣮ߦڥ
ػցֶशࢪࡦεΫϦϓτ Ҡ২Մೳ → γεςϜͷɾ݁߹͕༰қ → རศੑɾอकੑɹPythonͷ࠾ཧ༝ - طଘεΫϦϓτͱͷ૬ੑ
ɹจ໌ͷརثσʔλૢ࡞ͳΒPythonҰΤίγεςϜʹΔσʔλอଘͳΒBigQueryҰ GoogleʹཔΔΠϯϑϥΞϓϦ
ɹBQͷ࠾ཧ༝ɿ͍҆IUUQZBQDBTJBPSHUBMLTIPXDFBGBDFDBCEDB● Googleͷσʔληϯλʔ ɹεέʔϧϝϦοτʹΑΔ ɹίετ༏Ґ● ϢʔβʔͷҙݟΛͱʹ ɹྉۚϓϥϯΛݟͨ͠ܦҢ
ɹBQͷ࠾ཧ༝ɿૣ͍IUUQZBQDBTJBPSHUBMLTIPXDFBGBDFDBCEDB● ΧϥϜϕʔεઃܭʹΑΔ ɹΞΫηεߴԽ● େنσʔλ׆༻࣌ʹ ɹϘτϧωοΫͱͳΔ ɹDisk I/Oͷฒྻॲཧ
ɹBQͷ࠾ཧ༝ɿ҆৺GCPɾGsuiteͰݖݶཧ - ΈΜͳ͕͑ΔSQLΠϯλʔϑΣʔε - ෦ॲཧɾอकӡ༻Λؾʹͤͣʹ͑Δඪ४SQL ΫΤϦͷҠ২ੑ͕ߴ͍ ֶशίετ͕খ͍͞ ςετ͍͢͠पลπʔϧ ओཁBIπʔϧͳΒେܨ͕Δ pandas.io.gbq → Jupyterσόοά͕༰қ
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵύΠϓϥΠϯཧModel View
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵModel ViewऩूύΠϓϥΠϯཧ
ɹσʔλऩूݸਓใ ϚεΫࡁΈViewRecruit AdsConsoleRecruit Clash ReportόονετϦʔϛϯάWebAPIεΫϨΠϐϯάrequests +beautiful soup
ओཁͳDBɾϩά͔Βͷసૹ → ࠷খઃఆͰ࣮ݱՄೳɹσʔλసૹΛࢧ͑Δٕज़
Python (requests + beautiful soup) ʹΑΔ WebAPIίʔϧɾը໘εΫϨΠϐϯάɹJupyterͰૉૣ͘ࢼ͢·ͣϩʔΧϧڥͰࡁ·ͤΔ ܁Γฦ͠͏ͳΒεΫϦϓτԽIUUQTXXXTIBSFJDPOOFUEPDVNFOUpMFQZ
ΤϯδχΞ͕ੳʹܞΘ͍ͬͯͳ͍ͱEventσʔλ͕ϩάཁ͔݅Βൈ͚࿙Ε͕ͪ ෆཁσʔλͷաੵʹΑΔετϨʔδѹഭΛݒ೦ʢͦ͜ͰBQͰ͢Αʣɹσʔλੑ࣭ʹԠͨ͡ऩूํ๏4UBUFঢ়ଶσʔλ &WFOUཤྺσʔλྫ Ϣʔβʔͷ࠷ऴϩάΠϯ࣌ʢ্ॻ͖͞ΕΔʣ աڈͷϩάΠϯཤྺʢੵ͞ΕΔʣओͳ༻్ը໘දࣔ ʮ͜ͷਓલʹϩάΠϯ͠·ͨ͠Αʯੳ Ͳͷ͘Β͍ͷසͰϩάΠϯ͢Δ͔ڧΈɾಛϓϩμΫτຊମͷύϑΥʔϚϯε্ ͍͍ͪͪཤྺςʔϒϧͱ+PJOͯ͠ ࠷৽ϨίʔυΛ୳͢ͷ%#ෛՙσʔλੳͷ࠶ݱੑΛ୲อ ຖճϩάΠϯ࣌Λ্ॻ͖͞ΕͨΒ ੳͰ͖ͳ͘ͳͬͯ͠·͏ੳج൫Ͱͷ σʔλऩू࣍ͰҰׅஔ͖͑ %#ϩάˠࠩهˠ׆༻ͻͱඞཁࠩө લճҎ߱ͷϨίʔυ͚ͩΛநग़
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵModel ViewੵύΠϓϥΠϯཧ
3ߏ ࢀߟʰ10ઓ͑Δσʔλੳೖ - SQLΛثʹσʔλ׆༻࣌Λੜ͖ൈ͘ʱɹσʔλੵɹɹɹɹɹɹBigQuery - Google Cloud PlatformSource ݩσʔλͷίϐʔWarehouse ओཁࢦඪɾසग़σʔλ தؒςʔϒϧApp ֤πʔϧ͔Βࢀর ඇٕज़ऀ͚IFੵ
ྫɿ enmusubi__source__dbΞϯμʔείΞ2ͭͰཁૉΛ۠Δ BEMʢCSSͷઃܭख๏ʣϥΠΫͳنଇɹσʔληοτͷ໋໊نଇϓϩμΫτ໊ sourcewarehouseappϝΠϯDBαʔόϩάΞΫηεղੳπʔϧ
ɹσʔλࣙॻIUUQKQUFDIDSVODIDPNpWFCVJMEJOHCMPDLTPGBEBUBESJWFODVMUVSFར༻ऀσʔλϑΟʔϧυͱͦͷʢϝτϦΫεʣͷҙຯΛΔඞཁ͕͋Δɻ*ݩσʔλใͷϦϯΫूͷΈఏڙ γεςϜʢʹσʔλ༷ʣʑมԽ͢ΔͨΊ refs. ϩθολετʔϯ֤σʔλιʔεଆͰ༷ཧ e.g. ຊମDB → MySQL Workbench ɹERߏ/ίϝϯτDDLͰৗʹ࠷৽Խ
ൃల్্ͷϓϩμΫτɹ → ɹػೳՃͰසൟͳERมߋɹ → ɹBQਵ͍ͨ͠ɹੵσʔλʹϚΠάϨʔγϣϯ͕ඞཁ
ɹJupyter for Migration1. ϩʔΧϧͷJupyterͰ ɹϚΠάϨʔγϣϯεΫϦϓτΛॻ͘2. ͦͷ··JupyterͰ ɹ1ͷσʔλΛରʹಈ࡞֬ೝ3. PythonϑΝΠϧΛग़ྗ ɹ → αʔό্ͰશσʔλΛҠߦIUUQTXXXTIBSFJDPOOFUEPDVNFOUpMFQZ
※ͨͩ͠Python2.7੍͕͋ΔͷͰҰ෦ॻ͖͕͠ඞཁ → ύλʔϯ͕ݟ͖͑ͯͨΒมεΫϦϓτ● Φʔτεέʔϧ → Dataflow ɹGKEࢄ͚ͷઃܭ͕ඞཁ ● σʔλྔ͕গͳ͚ΕDatalab ɹJupyterͷίʔυ͕··͑ΔɹGCPͷαʔόΛར༻ɹ ● Googleઐ༻ઢͰBQΞΫηε ɹ ● ಉ͡ถࠃϦʔδϣϯʹཱͯΔɹBQͰ100ԯϨίʔυΛҠߦ͢ΔϘτϧωοΫᶃ N/W ϘτϧωοΫᶄ ϝϞϦ
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵModel ViewՃύΠϓϥΠϯཧ
3ߏ ࢀߟʰ10ઓ͑Δσʔλੳೖ - SQLΛثʹσʔλ׆༻࣌Λੜ͖ൈ͘ʱɹσʔλՃɹɹɹɹɹɹBigQuery - Google Cloud PlatformSource ݩσʔλͷίϐʔWarehouse ओཁࢦඪɾසग़σʔλ தؒςʔϒϧApp ֤πʔϧ͔Βࢀর ඇٕज़ऀ͚IFՃ Ճ
IUUQTHSPXUIIBDLKPVSOBMDPNLQJUSFFGPSBQQɹओཁࢦඪΛதؒςʔϒϧͰཧ
ɹPythonεΫϦϓτʹΑΔՃ
ɹsh → exec.py● γΣϧҾΛͱʹॲཧΛಈతৼΓ͚ ● ݸʑͷΤϯυϙΠϯτࡉԽ ɹˠ ΦʔτεέʔϧɺϦτϥΠɺಈ࡞֬ೝ͘͢͠
ɹController● ॲཧͷશମ૾Λهड़ ● Facadeͱͯ͠ଞॲཧΛݺͼग़͢
ɹModel● Ϗδωεࢦඪͱ1ର1ͰରԠ͢ΔΫϥε ● Create/ReadॲཧͷΈ → BQཧ ɹ → ϓϩάϥϜ্εςʔτϨεʹอͭ
ɹQuery● BigQueryͱͷR/WΛѻ͏SQL ● ςΩετॲཧͷࢧԉػೳΛ׆༻ ɹ → f-strings (3.6~)TypeHints (3.5~) ● ϝιουͱͯ͠Γग़͠ ɹ → ͷҾ͖͠ʹΑΔείʔϓ੍ޚ
ɹEnum● Enum(3.4~)Ͱσʔλݻ༗ͷίʔυΛཧ ● ຊମ (Java) ͷEnumΛҠ২ → Dictܕʹม ● ܧঝݩͷbaseEnumΛ༻ҙ → ڞ௨ͷσʔλऔಘॲཧΛ࡞
ɹResource● ֎෦γεςϜͱͷ࿈ܞॲཧ ● retryॲཧͷҰݩԽ → σʔλྔ͕૿͑ΔͱBQΑࣦ͘ഊ͢ΔͷͰ ● dotenvͰڥมΛಡΈࠐΉ → ຊ൪ɾ։ൃͰSlackνϟϯωϧΛ͚Δ
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵModel View׆༻ύΠϓϥΠϯཧ
ɹσʔλ׆༻ - PushܕIUUQGSFFJMMVTUSBUJPOTHBUBHOFUIUNMػցֶशࢪࡦ
ΞυϗοΫࣗಈԽɹσʔλ׆༻ - Pullܕ
ɹ୭ʹεΫϦϓτΛ༻ҙ͢Δ͔Yes Noͦͷ࡞ۀ ܁Γฦ͢ʁϞχλϦϯά ఆظతͳॲཧ ʲPushܕʳʢҰ෦ྫ֎͋Γʣεφοϓγϣοτ ΞυϗοΫͳੳɾௐࠪ ʲPullܕʳSQLΘ͔Δ ͻͱɾ෦ॺʁELTɿૄ௨ˠՃ ʢྫʣػցֶशΤϯδχΞ ʲPullܕʳETLɿՃˠૄ௨ SpreadsheetͰI/F ʲPushܕʳ※ՃࡁΈσʔλʹରͯ͠γϯϓϧͳSELECTจΛൃߦ͢Δ͚ͩͰࡁΉέʔεଟ͍
ɹDecorator● ֤View͚ͷσʔλՃʢग़ྗઌʹԠͨ͡มॲཧʹಛԽʣ ● ྫɿຖேSlackʹ͛ΔάϥϑΛ matplotlib Ͱඳը
ɹσʔλϑϩʔͷߏஙऩू Ճ ׆༻ੵModel ViewύΠϓϥΠϯཧ
ཧπʔϧʹظ͢Δཁ݅ʢཧʣ ɹσʔλύΠϓϥΠϯͷཧ1. ίʔυͰόʔδϣϯཧ 2. εέδϡʔϧࣗಈىಈ 3. ηοτΞοϓϝϯςφϯεͷָ͞(DBแͳͲ) 4. GUI/CUI྆ํͰϩάɾύϑΥʔϚϯεͷཧ 5. GUI/CUI྆ํͰมࢦఆͷ࠶࣮ߦ6. ࣗಈϦτϥΠॲཧɾεΩοϓػೳ 7. εΫϦϓτͷฒྻ࣮ߦ 8. αʔόͷΦʔτεέʔϧ 9. ༗ඇ८ճάϥϑͷࣗಈ࡞ 10. Python3.6~ରԠʢςΩετॲཧ͕֨ஈʹָʣ
ͷπʔϧΛܰ͘৮ͬͯࢼͯ͠ΈͨΓ ɹɹɹɹɹɹ ɹɹɹɹɹͲΕັྗత͚ͩͲҰேҰ → ͔Ώ͍ͱ͜Ζʹख͕ಧ͔ͳ͍ ɹɹɹɹɹɹ ● ίϯτϦϏϡʔτͷνϟϯε͔ʁ ɹɹɹɹɹɹ ● ਐ݄าͰັྗతͳͷͰ͍ͣΕҠ২͍ͨ͠ؾ࣋ͪ͋ΔɹWorkflow Engine
ͳΜ͔ͩΜͩͰ࠷ॳʹ͓खܰߏஙͨ͠ jenkins Λ͍ଓ͚͍ͯΔ ɹɹɹɹɹɹ● 2ܥ + BlueOcean + Pipeline of jenkinsfiles ɹɹɹɹɹɹ● ϦτϥΠػߏɾॲཧ࠷దԽଞʹྼΔ ɹɹɹɹɹɹ● jenkinsδϣϒ͔ΒγΣϧܦ༝Ͱ֤छॲཧΛίʔϧɹ҆ఆͷJenkins
ɹσʔλϑϩʔͷߏஙʢʣऩू Ճ ׆༻ੵModel ViewύΠϓϥΠϯཧ
ɹͲͷΑ͏ͳϓϩηεΛ࠾༻͔ͨ͠σʔλج൫
ʮσʔλج൫Λ࡞Δͧʯܭըࣦഊ͔Β࢝·ͬͨ
μογϡϘʔυΛ ෦ͷϞχλʔʹө͢ɹ࠷ॳͷҰา
1. ຖͷKPIΛ͢Β͢Βݴ͑ΔΑ͏ʹͳΔ͔ ɹ2. ҟৗʹؾ͍ͮͯਝʹಈ͚ΔΑ͏ʹͳΔ͔ɹߴ·Δظ
1िؒͰ୭ݟͳ͘ͳͬͨɹݱ࣮ඇͰ͋Δ
ɹɹɹᘳͳاըॻʁɹɹɹѹతͳ༧ࢉʁɹɹɹ1ؒͷ։ൃϓϩδΣΫτʁɹʮ࠷ڧͷσʔλج൫ʯΛࢧ͑Δϓϩηεʁ
ɹɹɹᘳͳاըॻʁɹɹɹѹతͳ༧ࢉʁɹɹɹ1ؒͷ։ൃϓϩδΣΫτʁɹʮ࠷ڧͷσʔλج൫ʯΛࢧ͑Δϓϩηεʁ❌
ɹɹ● ࢥ͍ͬͯͨͷͱҧͬͨɹɹ● ྑͦ͞͏͚ͩͲ݁ہΘͳ͔ͬͨɹ1ޙʹͭͷ
ɹʮԶͷߟ͑ͨ࠷ڧͷσʔλج൫ʯΘΕͳ͍IUUQXXXQSPKFDUDBSUPPODPN
→ ͜ΕͳΒ࣮֬ʹͬͯΒ͑Δɹগͣͭ͠ஔ͖͑Α͏ ࣮͏ͪͷ෦ॺͰຖேSlackʹάϥϑΛ खಈͰྲྀ͍ͯ͠ΔΜ͚ͩͲʂ ͜ͷػʹࣗಈԽ͠·͔͢ʂ
ɹɹɹQ.ɹ࠷ॳʹཁ݅Λશ෦ચ͍ग़ͤOKʁ ɹɹɹA.ɹNGʂʮΓ͍ͨ͜ͱʯมΘΔʂ ɹɹɹɹɹɹɹ ● ଞࣾϒϩάΛݟͯʮ͏ͪΓ͍ͨʂʯ ɹɹɹɹɹɹɹ ● ৽͍͠BIπʔϧͷఏҊʹʮ͜Ε͍͍ʂʯ ɹɹɹɹɹɹɹ ● ࣮ࡍʹը໘Λ࡞ͬͨΒʮͳΜ͔ҧ͏ʂʯ ɹɹ ɹɹ → มԽʹॊೈʹରԠ͢ΔʂɹΠςϨʔγϣϯΛճ͢͜ͱ͕େࣄ
TodoΛνέοτཧ → ॊೈʹ༏ઌॱҐΛೖΕସ͑Δɹνέοτۦಈ։ൃ
ɹλεΫͷ༏ઌॱҐ1 σʔλ͕·͕͍ͪͬͯΔ ٙͷௐࠪɹ1ͭؒҧ͍ͬͯΔͱશ෦৴༻Ͱ͖ͳ͘ͳΔͷͰ࠷༏ઌରԠɻ ɹ͙͢ௐࠪ͢Εؔऀͷ৴༻ߴΛ૿ͤΔɻ2 Modelվमɹσʔλ߲ՃͳͲɻ ɹଟগݟʹͯ͑͘͘Δঢ়ଶʹ͢Δ͜ͱ͕༏ઌɻ ɹ࠷ѱͷ߹ͰͲ͏ʹ͔͏ଆͰͰ͖ΔͷͰɻ3 Viewվमɹάϥϑͷݟӫ͑ͳͲɻ ɹҰ൪ʮมԽ͍ͯ͠ΔʯΠϯύΫτ → ؔߏஙʹد༩ɻ ɹ͍खʹͱͬͯॏཁͳͷͰՄೳͳൣғͰαϙʔτɻ4 γεςϜͷอकੑͱ ύϑΥʔϚϯεɹϘτϧωοΫʹͳͬͨΒରॲ͢Δɻ ɹ࠷ॳ͔ΒڽΔͱʮૣ͗͢Δ࠷దԽʯʹؕΓ͕ͪɻ ɹ࡞ۀνέοτʹ͢Δ·Ͱͳ͍ͭ͘Ͱʹ͢श׳Λ͚ͭΔɻ
ɹTest● unittest ʹΑΔࣗಈςετΛ׆༻ ● Excel༰Λظʹө → GreenʹͳΕ࠶ݱྃ - ·͞ʹ “TDD” ● SQLνϡʔχϯάϦϑΝΫλ࣌ʹσάϨΛଈ࣌ݕͰ͖ΔͷͰ૬ੑྑ͍
ɹॏްେͳExcelΛ࠶ݱ͢Δˠ·ͣղಡϚΠϯυϚοϓʹΑΔཧ εϥΠυʹೖΔΑ͏ʹʢೖΓ͖ͬͯͳ͍͚ͲʣॖΊͨΒը૾͕௵Εͯ͠͏ͨ……
ɹɹ1. Jupyter Notebook Ͱௐࠪ༰ͱ ɹɹɹσʔλΛ࠶ݱՄೳͳܗͰอଘ͢Δ ɹɹ2. Github Preview Λͦͷ·· ɹɹɹઆ໌ࢿྉͱͯ͠ར༻͢Δɹطଘͷܭࢉϛεσʔλෆ߹Λൃݟ
ɹχʔζ։ͷ॥֤෦ॺͷϢʔεέʔεΛ ώΞϦϯά͘͢͠ͳΔʮσʔλͷ͜ͱ͍ͭ͜ʹฉ͚ʯ ʹ ࣭૬ஊ͕དྷΔΑ͏ʹͳΔطଘσʔλͷϛεɺ ͞ΒͳΔհࡏ༨ʹؾ͚Δখ͞ͳ࣮ طଘσʔλͷܭࢉϛεෆ߹Λൃݟ → ใࠂͯ͢͠
ͱʹ͔͘ࣈ͕߹Θͳ͍ɹ॥Λࢧ͑ΔγεςϜ࿈ܞͷ్தͰ࢛ࣺޒೖ͕ੵΈॏͳΔExcel͕طʹ 4 x 1990 = 7959 ͱͳ͍ͬͯΔ࣮ߦλΠϛϯάͰूܭ݁Ռ͕มΘΔ → طଘͷ୭࠶ݱͰ͖ͳ͍
࣮ࡍʹඅͨ࣌ؒ͠ͷ80% ʹ ͻͨ͢Βಓͳ࡞ۀ ·͊ɺେมͰͨ͠ɻɹͱྦͷτϥϒϧγϡʔςΟϯά
ɹ֤෦ॺͷσʔλఏڙΛ࣮ݱσʔλج൫
ɹ͜Ε·ͰɿϏδωε෦ͷϝΠϯExcel● σʔλͷੵ → ॏ͍ϑΝΠϧ ● ؔ/ηϧͷଟॏࢀর → ௐࠪ/ղऍ͕ࠔ ● macOSͰ։͘ͱͳΜ͔յΕΔ։͘ͷʹ3ɻಈ͔ͦ͏ͱ͢Δͱ·ͨࢭ·ΔɻΤϯδχΞ͕ؾܰʹݟΒΕͳ͍ঢ়گ
ɹɹɹɹɹ ʴɹ͜Ε͔ΒɿTech + Culture = ValueςΫϊϩδʔ จԽɾϓϩηεσʔλΛ׆༻ͨ͠ ։ൃੈͷதʹސ٬ՁΛఏڙDone
ɹɹɹɹɹ ʴɹ͜Ε͔ΒɿTech + Culture = ValueςΫϊϩδʔ จԽɾϓϩηεσʔλΛ׆༻ͨ͠ ։ൃੈͷதʹސ٬ՁΛఏڙDone Todo
ɹϓϩμΫτ։ൃͱԾઆݕূBuildMeasureLearn
ɹϓϩμΫτ։ൃͱԾઆݕূBuildMeasureLearnͰ͖͍ͯΔͬͯΈΔ
ɹɹͲΜͳϢʔβʔ͕৽ػೳΛͬͨͷ͔ ɹɹϓϩμΫτʹͲΜͳظΛ͍ͬͯΔͷ͔ɹɹϏδωεKPIͲͷఔ্ͨ͠ͷ͔ ɹɹ෭࡞༻͔͋ͬͨʢ͋ΔػೳΛԡ͠ग़͢ → ผͷػೳ͕ΘΕͳ͘ͳΔՄೳੑʣɹ࣮ࢪͨ͠ࢪࡦͷ݁ՌΛੳIUUQHBUBHOFUGSFFJMMVTUSBJUJPO
ɹ͜Μͳײ͡ͰΓ·ͨ͠1. ϗϫΠτϘʔυʹੳཁ݅ ɹɹΧελϚʔߦಈભҠΛϕʔεͱͨ͠ࢦඪπϦʔ ɹɹϩάΠϯ্͕ → ΞΫγϣϯ͕… → ച্ࢉ ɹɹɹɹ2. JupyterͰੳ ɹɹɹɹɹίʔυશͯίϛοτཤྺͰཧ ɹɹɹɹɹɹɹɹ3. ੳ݁ՌΛؔऀʹڞ༗ ɹɹɹɹɹɹɹɹɹGithubͷϓϨϏϡʔը໘Λ׆༻
ɹKeepɹϏδωεࢹͰ։ൃΛݟΔࣗͷՌΛࣗͰݴ͑ΔΑ͏ʹͳͬͨ ʢྫɿ1ϲֻ݄͚ͨҊ݅Ͱച্xxԁ/ʣ
ɹProblemɹνʔϜͷվળ༨͕ݦࡏԽɹɹϝϯόʔ͕ࣗͷཧղෆʹڻ͘ ɹɹɹ ● ༷ - ͜ͷέʔεͩͱσʔλͷதͲ͏ͳΔʁ ɹɹɹ ● ϑϩϯτدΓϝϯόʔ͕ෳࡶͳSQLʹखؒऔΔ ɹɹɹ ● ͦͦϏδωεKPIΛѲ͍ͯ͠Δ͔ ɹɹɹ ● ͖ͪΜͱੳ͢ΔͨΊʹຊඞཁͩͬͨϩάཁ݅ͷ࿙Ε
ɹTryɹϓϩμΫτࢤνʔϜͷୈҰาݴΘΕͨͷΛ࡞Δ͚ͩʢBuildॏࢹʣࢥߟ͔Βͷ٫ʮ͞ΒʹࣈΛ৳͢ʹͲ͏͢ΔʁʯࢪࡦΛىҊ͢ΔΤϯδχΞ͕ग़࢝ΊΔϏδωε෦ͱนଧͪ 㲗 ࢼߦࡨޡ αϯϓϧ࣮ɹษڧձͰڝ߹ͷΤϯδχΞΛั·͑ͯΛฉ͘ ࣾؔऀʹҙࣝΛώΞϦϯάɹࣈΛͱʹޮՌݟཱͯνʔϑϓϩσϡʔαʔ͔ΒGoαΠϯ
ɹ͍͔ʹਁಁͤ͞Δ͔BuildMeasureLearnͰ͖͍ͯΔͬͯΈΔ ਁಁͤ͞Δ
͋ͷख͜ͷखͰʮσʔλੳΖ͏ʯͷۭؾΛৢɹ ● σʔλੳϋοΧιϯͷ։࠵ɹ ● ே׆ x ࣗ༝ݚڀʢϏοτίΠϯͷ্͕ΓཁҼΛ౷ܭղੳ͢Δۚͷऀ͕େྔൃੜʣɹ ● JupyterHubΛ֦ுͨ͠PythonֶशαΠτ ɹɹ֎ʹஔ͚ͳ͍σʔλ͚ͷnbviewerΛࣾఏڙɹϝοηʔδϯά
1ͭͷϞχλʔΛғΜͰશһͰ࡞ۀ͢Δɹɹ ● पғ͕ௐͨΓΞυόΠε͠ͳ͕ΒਐΊΔ ɹɹɹ → ϋϚΒͳ͍ / ࠳ંΛ͙ɺTipsίπΛڞ༗͋͠͏ ɹɹ ● օ͕ΔͳΒࣗΔ͔ʂͷޙԡ͠ ɹɹɹ → ʮͬͯΈͨΒࢥͬͨҎ্ʹྑ͔ͬͨʯͷମݧɹϞϒ ϓϩάϥϛϯά σʔλੳ
σʔλ༷ʹৄ͍͠ʢগͳ͘ͱௐࠪ͢ΔεΩϧ͍ͬͯΔʣΤϯδχΞ͕ ୲ྖҬΛ͛Δ͜ͱͰલޙఔͷϦʔυλΠϜΛॖ ɹϓϩηεண - ։ൃఔʹΈࠐΉઃܭ ࢼݧ ϦϦʔεޮՌ ଌఆཁ݅ ఆٛཁٻ ཧઃܭ ࢼݧ ϦϦʔεޮՌ ଌఆཁ݅ ఆٛཁٻ ཧطʹ͍ͬͯΔੳཁٻ ϩάཁ݅ ܭଌΞΠσΞ
ΤϯδχΞ ʹ ͦͷͰσʔλ༷ΛௐΔ͜ͱ͕Ͱ͖Δ σʔλαΠΤϯε෦͕༷ώΞϦϯάΛॏͶͯ 1ि͔͚͍ؒͯͨੳ͕1࣌ؒͰྃʢϑϩʔޮԽʣɹ࣮ɿΞδϦςΟͷ্ϝΠϯϞχλʔ Jupyter NotebookαΠυϞχλʔ ϓϩμΫτͷιʔείʔυʴ
ɹϓϩμΫτ։ൃͱԾઆݕূBuildMeasureLearnͰ͖͍ͯΔͬͯΈΔ Ͱ͖͍ͯΔ
ɹLearnࠓޙͷ՝ ·ͩࢼߦࡨޡதͰ͢
ɹؾ͖ - “Ծઆ” ΛΒͳ͍ͱ “ݕূ” Ͱ͖ͳ͍IUUQXXXOJLLFJCQDPKQBUDMQVCNLUCPPL1Ϗδϣϯઓུઓज़
“Ծઆ” Λνʔϑϓϩσϡʔαʔʹ࣭↓ ߟ͑Λฉ͍ͨΤϯδχΞʮ͍ɺԶ͜͏ࢥ͏ʂʯ↓ ϒϥϯυ࠶ݕ౼ϫʔΫγϣοϓͷ։࠵ ͲΜͳސ٬ʹͲΜͳՁΛఏڙ͢Δͷ͔ ෦ॺΛԣஅ͓ͯ͠ޓ͍ͷߟ͑ΛͿ͚ͭ߹͏ɹରͷ׆ੑԽ
֤ࣗͷҙݟ͕ʮ͋ͳͨͷײͰ͢ΑͶʯʮͳΜ͔σʔλͱ͔͋ΔΜͰ͔͢ʯঢ়ଶɹҙݟͷͿ͚ͭ߹͍ → ݟ͖͑ͯͨ՝ࣗͨͪސ٬ͷ͜ͱΛ ຊʹཧղ͍ͯ͠Δͷ͔ʁউखͳࢥ͍ࠐΈͰ ϓϩμΫτΛ ։ൃ͍ͯ͠ͳ͍͔ʁ
ɹϦΫϧʔτͷྺ࢙Ӧۀ͕ళฮʹग़͍ͯ σʔλΛऩू͍ͯͨ͠σʔλʢࣄ࣮ʣΛ࣋ͪدͬͯ ෦ॺؒͰҙݟΛͿ͚ͭ߹͏ࢢͷ࠶ఆٛ ৽͍͠ϏδωεՁͷߏங ྫɿঁੑ͚ब৬ϝσΟΞ → ࣾձਐग़ͷࢧԉ ྫɿΫʔϙϯࡶࢽ → ༧αΠτ ͷస[email protected] [email protected] IUUQXXXJSBTVUPZBDPNOFXQPQIUNM
4&$*Ϟσϧɹσʔλ͕ࢧ͑Δࣝސ٬Ձ উͪےɾUVPΤϯδχΞ ू٬σβΠϯ ηΩϡϦςΟ๏ΧελϚʔαϙʔτ ใࢧԉऀɾ։ऀͱͯ͠ͷ σʔλαΠΤϯςΟετσʔλ
શͯ ސ٬ཧղ ͔Β࢝·ΔσʔλͦͷͨΊͷஅࡐྉ
Ӧۀ͕ΛͬͯσʔλΛूΊΔ ΫϥΠΞϯτͱపఈతʹ͖߹͍ʮউͪےʯΛݟ͚ͭΔ ͦ͏ͬͯϦΫϧʔτͷ ͜Ε·Ͱͷ50Λ࡞͖ͬͯͨ ϝσΟΞΛ௨ͯ͠ੈքΛ࡞͖ͬͯͨ
ΤϯδχΞ͕खΛͬͯσʔλΛूΊΔ ΧελϚʔͱపఈతʹ͖߹͍ʮUVPʯΛݟ͚ͭΔ ͦ͏͢Δ͜ͱ͕ϦΫϧʔτͷ ͜Ε͔Βͷ50Λ࡞Δͣ ϓϩμΫτΛ௨ͯ͠ੈքΛ࡞Δͣ
ϦΫϧʔτʹݶͬͨͰͳ͍ ͜Ε͔ΒͷੈքΛ࡞Δͷɺ ୭ΑΓ ͱ ސ٬ ʹਫ਼௨ͨ͠ ΤϯδχΞͷׂ Ͱ͢ ͍·͜ͷʹ͍Δ ࢲͨͪࣗ Ͱ͢ σʔλ׆༻ɺͦͷୈҰาͩͱࢥ͍ͬͯ·͢
σʔλΛ׆༻͢ΔͨΊʹΓӽ͑Δน● େن৫ͰͲͷΑ͏ͳγεςϜɾϓϩηεΛ࠾༻͢Δ͔ʁ ● BuildॏࢹݱʹͲ͏ͬͯMeasureɾLearnΛண͢Δ͔ʁ1ͭͷϓϩμΫτʹ͓͚ΔࣄྫΛڞ༗͠·ͨ͠ɹຊͷৼΓฦΓ
σʔλ׆༻ͷਪਐʢදͱͯ͠ൃද͠·͕࣮ͨ͠ࡍʹ൴ΒͷޭʹΑΔͱ͜Ζ͕େ͖͍Ͱ͢ʣɹNaoya OsugiɹLee HeesungɹTatsuya IshibeɹTakuya BeppuɹTakahiro Kato ɹKohei AotaɹRyo InoueɹHiroki SakamotoɹShinryo UchidaɹShinsaku Kouno ɹShin KanouchiɹGeeWook KimɹKoh Fujiwara ٕज़ΞυόΠεɾ৫ࢧԉɹSho ItoɹShinya NishinakaɹNaoki Ainoya ɹKeisuke SoneɹSatoshi UejimaɹNorihisa MiyakawaൃදαϙʔτɹShun OnoɹYusaku TokunagaɹKazutaka SakuraiɹSaki Kato※ࢿྉɾ༰ʹޡΓෆదͳදݱ͕͋ΕൃදऀͷϛεͰ͢ɻ@yuzutas0 ʹ͝࿈བྷ͍͚ͨͩΔͱ͍Ͱ͢ɻɹSpecial Thanks - ج൫ͷಓ1ਓʹͯ͠Βͣ
গ͠Ͱଟ͘ͷ ։ൃݱ ͕ ੈͷதʹ ྑ͍ϓϩμΫτ Λఏڙ͢Δ ώϯτ ʹͳΕͨΒͱࢥ͍ͬͯ·͢ɹ͍͑ͨ͜ͱ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠presented by @yuzutas0 https://www.pexels.com/photo/close-up-of-computer-keyboard-257949/ɹhttps://www.pexels.com/photo/technology-computer-lines-board-50711/ɹhttps://www.pexels.com/photo/black-and-white-business-chart-computer-241544/