データを中心とするWebサービスの 開発・運用を実現する技術

2018/6/23 【CAMPHOR x LIVESENSE】 Webサービスの開発・運用におけるデータエンジニアリング

Transcript

1. σʔλΛத৺ͱ͢ΔWebαʔϏεͷ ։ൃɾӡ༻Λ࣮ݱ͢Δٕज़ ాத ঵ଠ࿠ (@yubessy) CAMPHOR x LIVESENSE WebαʔϏεͷ։ൃɾӡ༻ʹ͓͚ΔσʔλΤϯδχΞϦϯά
2. About me ాத ঵ଠ࿠ (@yubessy) ςΫϊϩδΧϧϚʔέςΟϯά෦ σʔλϓϥοτϑΥʔϜάϧʔϓ • 2010 - 2014 ژ౎େֶ޻ֶ෦৘ใֶՊ ܭࢉػՊֶίʔε • 2014 - 2016 ژ౎େֶେֶӃ৘ใֶݚڀՊ ࣾձ৘ใֶઐ߈ • 2016 - גࣜձࣾϦϒηϯε • σʔλ෼ੳج൫ Livesense Analytics ͷ։ൃɾӡ༻ • ػցֶशج൫ Livesense Brain ͷ։ൃɾӡ༻
3. Works • ࣄۀʹدΓఴ͏σʔλج൫ͷҭͯํ • ػցֶशγεςϜͷ৴པੑΤϯδχΞϦϯά • σʔλ෼ੳΛࢧ͑ΔʮศརΧϥϜʯͷ໰୊఺ͱͦͷղܾࡦ • Ϛϧνίϯςφߏ੒ʹΑΔػցֶशΞϧΰϦζϜͱΞϓϦέʔγϣϯͷૄ݁߹Խ • ػցֶशγεςϜͷӡ༻՝୊ͱίϯςφΦʔέετϨʔγϣϯ͕΋ͨΒ͢΋ͷ • Apache Sparkͷ3ͭͷAPI: RDD, DataFrame͔ΒDataset΁ • Facebookͷߴ଎ϕΫτϧܭࢉϥΠϒϥϦfaissΛϨίϝϯυAPIʹ࢖ͬͯΈΔ
4. ࠓ೔఻͍͑ͨ͜ͱ αʔϏε։ൃͰͳͥσʔλ͕ॏཁʁ αʔϏεͷσʔλ׆༻͸Ͳ͏͋Δ΂͖ʁ σʔλʹؔΘΔ࢓ࣄʹٻΊΒΕΔ͜ͱ͸ʁ
5. ͦͷલʹ
6. ͳΜͷͨΊʹαʔϏεΛͭ͘Δͷ͔ʁ
7. αʔϏεΛͭ͘Δཧ༝ Ϣʔβͷ໾ʹཱͬͯɺ͓ۚΛ΋Β͏
8. ϦϒηϯεͷαʔϏε͸ Ͳ͏΍ͬͯϢʔβͷ໾ʹཱ͍ͬͯΔʁ Ͳ͏΍͓ͬͯۚΛ΋Β͍ͬͯΔʁ
9. Ωʔϫʔυ : ੒Ռใुܕ
10. ͨͱ͑͹ : ϚοϋόΠτͷ͘͠Έ ܝࡌ՝ۚ (ଞࣾαʔϏε) 1. ΞϧόΠτΛޏ͍͍ͨ 2. ٻਓΛܝࡌ ← අ༻ൃੜ ࠾༻՝ۚ (ϚοϋόΠτ) 1. ΞϧόΠτΛޏ͍͍ͨ 2. ٻਓΛܝࡌ ← ແྉ 3. Ԡืऀ͕࠾༻͞Εͯॳग़ۈ ← අ༻ൃੜ
11. ੒ޭใुܕ = αʔϏεͷར༻։࢝Ͱ͸ͳ͘໨తୡ੒ͷ࣌఺ͰใुΛ΋Β͏
12. ੒ޭใुܕͷϝϦοτ ࠾༻͠ͳ͚Ε͹අ༻͸ൃੜ͠ͳ͍ = اۀͷ͓͕ۚແବʹͳΒͳ͍
13. σϝϦοτ͸ʁ
14. ੒ޭใुܕͷσϝϦοτ ࠾༻͠ͳ͚Ε͹අ༻͸ൃੜ͠ͳ͍ = ܝࡌ͕Ϧϒηϯεͷച্ʹͳΒͳ͍
15. Ͳ͏͢Δʁ
16. Ϧϒηϯε͕΍Δ΂͖͜ͱ ܝࡌ͚ͩͰͳ͘Ԡืɾ࠾༻·Ͱ੒ޭͤ͞Δ = ϢʔβͱاۀΛ͏·͘Ϛονϯά͢Δ
17. Ͳ͏΍ͬͯʁ
18. Ϧϒηϯε͕஫ྗ͍ͯ͠Δ͜ͱ σʔλΛ׆༻͢Δ
19. ఻͍͑ͨ͜ͱ1 αʔϏε։ൃͰͳͥσʔλ͕ॏཁʁ
20. σʔλͱฉ͍ͯɾɾɾ • Ϗοάσʔλ, ϦΞϧλΠϜσʔλ, .... • σʔλ෼ੳ, σʔλαΠΤϯε, .... • ਓ޻஌ೳ, ू߹஌, .... • ػցֶश, ࣗવݴޠॲཧ, ... • σΟʔϓϥʔχϯά, ... • Hadoop, Spark, ...
21. Ұ୴๨Ε͍ͯͩ͘͞
22. ͦ΋ͦ΋ WebαʔϏε͕ѻ͏σʔλͬͯʁ
23. WebαʔϏε͕ѻ͏σʔλ ίϯςϯπ : αʔϏε͕Ϣʔβʹఏڙ͢Δ৘ใ • ΞϧόΠτٻਓථ • ΞϧόΠτઌͷΫνίϛ Ϣʔβσʔλ : αʔϏε͕Ϣʔβ͔Βड͚औΔ৘ใ • ر๬৚݅ (৬छ, ۈ຿஍, ...ʣ • ݕࡧͷΩʔϫʔυ • ϖʔδͷΞΫηεϩά
24. Ϣʔβσʔλ͸Ͳ͏໾ཱͭʁ
25. Ϣʔβσʔλͷొ৔ ੲͷWeb : ϦϯΫΛͨͲͬͯࣗྗͰίϯςϯπʹͨͲΓண͘ • αΠτϚοϓɾϦϯΫू • Yahoo! σΟϨΫτϦݕࡧ Google ݕࡧͷొ৔ • ίϯςϯπʹϦϯΫΛషΕ͹ Google Bot ͕ΫϩʔϦϯά • Ϣʔβ͸΄͍͠৘ใΛ Google ʹݕࡧΩʔϫʔυͱͯ͠༩͑Δ
26. Ϣʔβσʔλ΁ͷ஫໨ Ϣʔβ͕༷ʑͳσʔλΛఏڙ͢ΔΑ͏ʹͳͬͨ • Google, Twitter, Instagram, LINE, ... σʔλ׆༻ͷͨΊͷٕज़͕ൃୡͨ͠ • େྔσʔλͷ஝ੵ : ෼ࢄॲཧ, Ϋϥ΢υ, ... • ༗༻ͳ৘ใͷநग़ : ػցֶश, ࣗવݴޠॲཧ, ... → ϢʔβσʔλΛར༻ͯ͠ίϯςϯπΛ͏·͘ಧ͚͍ͨ
27. ϦϒηϯεͰ͸Ͳ͏࢖͍ͬͯΔʁ
28. ͨͱ͑͹ Ձ஋ͷߴ͍ٻਓΛݟ͚ͭ΍͘͢͢Δ
29. (࠶ܝ) ϚοϋόΠτͷ͘͠Έ ܝࡌ՝ۚ (ଞࣾαʔϏε) 1. ΞϧόΠτΛޏ͍͍ͨ 2. ٻਓΛܝࡌ ← අ༻ൃੜ ࠾༻՝ۚ (ϚοϋόΠτ) 1. ΞϧόΠτΛޏ͍͍ͨ 2. ٻਓΛܝࡌ ← ແྉ 3. Ԡืऀ͕࠾༻͞Εͯॳग़ۈ ← අ༻ൃੜ
30. ʮՁ஋ͷߴ͍ٻਓʯͱ͸ʁ
31. Ϣʔβ : ࠾༻͞Ε΍͍͢ اۀ : Ԡื͞Ε΍͍͢ ϚοϋόΠτ : Ԡื & ࠾༻͞Ε΍͍͢
32. ʮՁ஋ͷߴ͍ٻਓʯΛͲ͏΍ͬͯ஌Δʁ
33. ʮՁ஋ͷߴ͍ٻਓʯͷࢦඪ Ԡื཰ɾ࠾༻཰
34. Ԡื཰ͱ࠾༻཰ Ԡื཰ = Ԡื਺ / Ӿཡ਺ ࠾༻཰ = ࠾༻਺ / Ԡื਺ ٻਓͷՁ஋ ∝ Ԡื཰ ✕ ࠾༻཰ • Ԡื཰͕ߴ͍ → Ϣʔβʹਓؾ͕͋Δ • ࠾༻཰͕ߴ͍ → اۀ͕ਓΛޏ͍͍ͨ
35. ӾཡɾԠื͕·ͩθϩͳΒʁ
36. ৽ணٻਓͷԠื཰ɾ࠾༻཰͸ܭࢉͰ͖ͳ͍ → աڈσʔλ͔Βػցֶश
37. Ԡื཰ɾ࠾༻཰ͷ༧ଌ • ΞϧΰϦζϜ : ϩδεςΟοΫճؼ • ಛ௃ : ৬छ, ۈ຿஍, ళฮ਺, ٳ೔, γϑτ਺, ... • ڭࢣ : աڈͷଟ਺ͷٻਓͷӾཡɾԠืɾ࠾༻ ← Ϣʔβσʔλ (ߦಈϩά)
38. ༧ଌͨ͠஋ΛͲ͏࢖͏ʁ → ݕࡧ݁ՌͷϥϯΩϯά
39. ݕࡧ݁ՌͷϥϯΩϯά 1. ϢʔβͷݕࡧΩʔϫʔυʹϚον͢ΔٻਓͷϦετΛऔಘ 2. ༧Ί༧ଌ͓͍ͯͨ͠Ԡื཰ɾ࠾༻཰ʹج͍ͮͯϦετΛฒͼସ͑ public void addDocs(TopFieldDocs mainDocs, int reSortDocsNum) throws JsenPageLimitExceedingException, JsenInvalidReSortUnitException { int sortUnitIndex = getIndexOf(dhondtUnit, mainDocs.fields); this.docCountTable = new DocCountTable(mainDocs.fields, reSortUnits, reSortDocsNum); java.util.ListIterator<FieldDoc> iter = fieldDocsIter(mainDocs.scoreDocs, reSortDocsNum); HashMap<FieldDoc, Float> reSortedDocs = new HashMap(reSortDocsNum); while (iter.hasNext()) { FieldDoc doc = iter.next(); float score = ((Integer) (doc.fields[sortUnitIndex])).floatValue(); int count = docCountTable.getCountOf(doc); reSortedDocs.put(doc, score / (1 + count * dhondtCoef)); docCountTable.add(doc); } ArrayList<Entry<FieldDoc, Float>> priorityList = new ArrayList<>(reSortedDocs.entrySet()); Collections.sort(priorityList, (obj1, obj2) -> obj2.getValue().compareTo(obj1.getValue())); this.allDocs = priorityList.stream().map(Entry::getKey).toArray(FieldDoc[]::new); }
40. Q. େม͡Όͳ͍ʁ
41. A. େมͰ͢
42. σʔλ׆༻ͷ೉͠͞ ͜ͷγεςϜ͸ൺֱత୯७ͳ෦ྨ ͦΕͰ΋େมͳ͜ͱ͸ͨ͘͞Μ͋Δ • ͲΜͳΞϧΰϦζϜɾಛ௃ΛબͿ͔ʁ • ӾཡɾԠืɾ࠾༻σʔλΛͲ͔͜ΒऔΔ͔ʁ • ༧ଌ஋ΛݕࡧγεςϜʹͲ͏औΓࠐΉ͔ʁ • ݕࡧ݁ՌͷιʔτͷܭࢉྔΛͲ͏ݮΒ͔͢ʁ • ͜Ε͕ຊ౰ʹϢʔβ΍اۀʹ໾ʹཱͭͷ͔ʁ
43. ͳΜͰ೉͍͠ͷʁ
44. σʔλ׆༻Λ೉͘͢͠ΔݪҼ γεςϜͷෆ࣮֬ੑ͕ߴ͍ ༷ʑͳઐ໳ੑ͕ඞཁʹͳΔ
45. γεςϜͷෆ࣮֬ੑ͕ߴ͍ ී௨ͷ৔߹ = σʔλ͕มΘͬͯ΋ϩδοΫ͸มΘΒͳ͍ • ϩδοΫ͸ίʔυ͚ͩͰܾ·Δ • ίʔυΛݟΕ͹ೖྗʹର͢Δग़ྗ͸Θ͔Δ ػցֶश = σʔλ͕มΘΔͱϩδοΫ͕มΘΔ • ಉ͡ίʔυͰ΋ֶशσʔλ͕ҧ͑͹ผͷϞσϧ • Ϟσϧͷύϥϝʔλ͸ਓ͕ؒཧղ͠ʹ͍͘
46. γεςϜͷෆ࣮֬ੑ͕ߴ͍ → αʔϏεΠϯ͢Δ·Ͱ݁Ռ͕ݟ͑ͳ͍
47. ༷ʑͳઐ໳ੑ͕ඞཁʹͳΔ యܕతͳWeb։ൃ • αʔόαΠυ, ϑϩϯτ, Πϯϑϥ, ΞϓϦ σʔλΛ׆༻ͨ͠Web։ൃ • αʔόαΠυ, ϑϩϯτ, Πϯϑϥ, ΞϓϦ • ػցֶशΤϯδχΞ, σʔλΤϯδχΞ • ΞφϦετ, σʔλαΠΤϯςΟετ
48. ༷ʑͳઐ໳ੑ͕ඞཁʹͳΔ → ਓࡐͷෆ଍ɾ։ൃίετͷ૿େ
49. Α͋͘Δࣦഊ
50. Ғ͍ਓ ʮਓ޻஌ೳͰ͍͍ײ͡ͷ੒ՌΛग़ͯ͘͠Εʯ
51. σʔλΤϯδχΞ (Λ೚͞Εͯ͠·ͬͨਓ) ʮΞΫηεϩά͸શ෦͋Γ·͢ʂʂʂʯ
52. ݱ࣮ ϑΝΠϧαʔόʹཷ·ͬͨ਺ສͷϑΝΠϧ /var/log/apache.log-2012-05-19.gz ͳ͓ɺ͋Δ೔͔ΒಥવϑΥʔϚοτ͕มΘΔ
53. σʔλαΠΤϯςΟετ (Λ೚͞Εͯ͠·ͬͨਓ) ʮ͜ͷΞϧΰϦζϜͰਫ਼౓͕100%ʹʂʂʂʯ
54. ݱ࣮ ֶशσʔλ͸40݅ ֶश΋ςετ΋ಉ͡σʔλ ௐ΂ͨΒܾఆ໦ΞϧΰϦζϜ͕աֶश͍ͯͨ͠
55. ͜͏ͳΒͳ͍ͨΊʹ
56. ఻͍͑ͨ͜ͱ2 αʔϏεͷσʔλ׆༻͸Ͳ͏͋Δ΂͖ʁ
57. ͦͷલʹ : େมͳΒ΍Βͳͯ͘Α͍ͷͰ͸ʁ
58. σʔλ׆༻͸େม͔ͩΒ΍Βͳͯ͘Α͍ʁ ৑ஊͰ͸ͳ͘ɺ͜Ε΋Ұͭͷཱ೿ͳબ୒ σʔλΛ࢖Θͳͯ͘΋αʔϏε͕͏·͍͘͘ͳΒͦΕͰ͍͍ σʔλ͕ແ͍ͷʹσΟʔϓϥʔχϯάͳͲʹखΛग़ͯ͠΋࢓ํͳ͍ ٯʹݴ͑͹ɺσʔλ׆༻Λ΍͍ͬͯ͘ʹ͸ͦΕͳΓͷ౒ྗ͕ඞཁ ͦΕͰ΋ɺσʔλ׆༻ͰαʔϏε͕ΑΓྑ͘ͳΔͳΒ΍Δ΂͖
59. ղܾ͢΂͖՝୊ ෆ࣮֬ੑΛ͏·͘ѻ͏ ઐ໳ੑΛ੒Ռʹͭͳ͛Δ
60. ෆ࣮֬ੑΛ͏·͘ѻ͏ʹ͸ʁ
61. ෆ࣮֬ੑΛ͏·͘ѻ͏ʹ͸ʁ → ҆શɾޮ཰తͳ։ൃͱݕূΛՄೳʹ
62. γεςϜͷෆ࣮֬ੑΛѻ͏ํ๏ ෆ࣮֬ੑ͸෭࡞༻Ͱ͸ͳ͘ɺຊ࣭తͰආ͚ΒΕͳ͍΋ͷ ෆ࣮֬ੑΛ཈੍͢ΔͷͰ͸ͳ͘ɺ্ख͘ѻ͑ΔΑ͏ʹ͢Δ • σʔλ͕มΘΔͱϞσϧ΋มΘΔ → ։ൃͰ΋ຊ൪ͱಉ͡σʔλΛ࢖͏ • σʔλநग़΍੔ܗʹख͕͔͔ؒΔ → σʔλͷ৔ॴͱલॲཧΛҰݩԽ → ͜ΕΛՄೳʹ͢Δσʔλ෼ੳج൫Λͭ͘Δ
63. Ϧϒηϯεͷσʔλ෼ੳج൫ ௨শ : Livsense Analytics • ϚοϋόΠτɾస৬ձٞɾస৬φϏͳͲɺ֤αʔϏεͷσʔλΛू໿ • ίϯςϯπ (ٻਓ৘ใ౳) ͱϢʔβσʔλ (ߦಈϩά౳) ͷ݁߹෼ੳΛՄೳʹ όοΫΤϯυ : Amazon Redshift • ෼ੳ༻్ʹಛԽͨ͠ྻࢤ޲ܕ෼ࢄσʔλ΢ΣΞϋ΢ε • ϨΠςϯγΑΓεϧʔϓοτΛॏࢹͨ͠ΫΤϦΤϯδϯ • 500+ ςʔϒϧ / 100ԯ+ Ϩίʔυ / 100+ ࣾ಺Ϣʔβ
64. Livesense Analytics ͷશ๴
65. ೔ʑ౤͛ΒΕ͍ͯΔΫΤϦ SELECT DATE(n_cpl.pull_at) AS dt, CASE WHEN n_cpl.pull_condition_id = 47 THEN '1' WHEN n_cpl.pull_condition_id = 48 THEN '2' WHEN n_cpl.pull_condition_id = 49 THEN '3' ELSE 'other' END AS list, COUNT(DISTINCT n_c.id) AS contacts, COUNT(DISTINCT CASE WHEN service_start.contact_id IS NOT NULL THEN n_c.id ELSE NULL END) AS agreements, COUNT(DISTINCT CASE WHEN application_proxy.summarizable_id IS NOT NULL THEN n_c.id ELSE NULL END) AS agencies, COUNT(DISTINCT CASE WHEN cv.id IS NOT NULL THEN n_c.id ELSE NULL END) AS conversions FROM nkd.contact_pull_logs AS n_cpl INNER JOIN nkd.contacts AS n_c ON n_cpl.contact_id = n_c.id LEFT JOIN ( SELECT contact_id, staff_id, action_at FROM nkd.histories AS n_h INNER JOIN nkd.histories_history_actions AS n_hha ON n_h.id = n_hha.history_id WHERE n_hha.history_action_id = 1 AND n_h.history_result_id = 1 AND n_h.created_at >= '2017-12-19' ) AS service_start ON n_c.id = service_start.contact_id LEFT JOIN ( SELECT * FROM nkd.kpi_raw_data WHERE name = 'application_proxy' AND created_at >= '2017-12-19' ) AS application_proxy ON n_c.member_id = application_proxy.member_id LEFT JOIN ( SELECT id FROM nkd.applications WHERE recruit_state IN ('hire','pre_hire_confirm_admin') ) AS cv ON application_proxy.summarizable_id = cv.id WHERE n_cpl.pull_at >= '2017-12-19' GROUP BY dt, list ;
66. ઐ໳ੑΛ੒Ռʹͭͳ͛Δʹ͸ʁ
67. ઐ໳ੑΛ੒Ռʹͭͳ͛Δʹ͸ʁ → ٕज़બ୒ͷࣗ༝౓Λ޲্ͤ͞Δ
68. σʔλ׆༻ٕज़ͷଟ༷ੑ ՝୊ɾ໨తɾಘҙ෼໺ʹΑͬͯద͢Δٕज़͸ҟͳΔ ຊདྷ͸֤͕ࣗ100%ͷྗΛൃشͰ͖Δٕज़Λ࢖͍͍ͨ • Web։ൃ : Ruby etc. 㲗 σʔλ׆༻ : Python etc. • σʔλ׆༻Ͱ΋ R, Python, Julia ͳͲ͸Ұ௕Ұ୹ → ॊೈͳٕज़બ୒͕Ͱ͖Δ࣮ߦ؀ڥΛ੔උ͢Δ
69. Ϧϒηϯεͷػցֶशج൫ ௨শ : Livsense Brain • ༧ଌɾϨίϝϯυͳͲ֤αʔϏεͷػցֶशγεςϜΛ࣮ߦ • ؀ڥΛίϯςφԽ͠ɺ༷ʑͳγεςϜΛ؆୯ʹσϓϩΠ όοΫΤϯυ : Google Kubernetes Engine • ෳ਺ίϯςφͷ࣮ߦ΍ڠௐΛՄೳʹ͢ΔΦʔέετϨʔγϣϯαʔϏε • ҟͳΔγεςϜͰ΋ϦιʔεΛڞ༗ɾ༗ޮ׆༻ • 16+core CPU / 100+GB RAM / 15+छྨ ίϯςφ
70. Livesense Brain ͷશ๴
71. R / Shiny ʹΑΔޮՌ༧ଌWebΞϓϦ
72. Julia ʹΑΔϨίϝϯυΞϧΰϦζϜ࣮૷ function bpmf( users::Vector{Int}, items::Vector{Int}, rates::Vector{Float64}, N::Int, M::Int, D::Int, n_sample::Int) rateidx = sparse(users, items, 1:length(rates)) U = zeros(n_sample, N, D) V = zeros(n_sample, M, D) for t in 1:(n_sample-1) SS = (1. / N) * ΣXXt(U[t, :, :]) SU = (1. / N) * sum(U[t, :, :], 1)[:] W0ast = inv(inv(W0) + N * SS + (β0 * N) / (β0 + N) * (μ0 - SU) * (μ0 - SU)') λ_u = rand(Wishart(ν0 + N, symmetric(W0ast))) μ0ast = (β0 * μ0 + N * SU) / (β0 + N) μ_u = rand(MvNormal(μ0ast, symmetric(inv((β0 + N) * λ_u)))) ... for i in 1:N js = find(rateidx[i, :]) SV_i = ΣXXt(V[t, js, :]) λast_inv = inv(λ_u + α * SV_i) SVR_i = sum(V[t, js, :] .* rates[rateidx[i, js].nzval], 1)[:] μast = λast_inv * (α * SVR_i + λ_u * μ_u) U[t+1, i, :] = rand(MvNormal(μast, symmetric(λast_inv))) end ... end return U, V end
73. Q. ΍ͬͺΓେม͡Όͳ͍ʁ
74. A. ͦΕͳΓʹେมͰ͢
75. େม͚ͩͲɾɾɾ ϦϒηϯεͰ͸ɺ͜Ε·Ͱଟ͘ͷࢼߦࡨޡΛੵΈॏͶΔதͰ σʔλ׆༻ʹΑΓϢʔβʹ໾ཱͭαʔϏεΛ࡞ΕΔΑ͏ʹͳ͖ͬͯͨ ߴ͍ઐ໳ੑΛۦ࢖ͯ͠αʔϏεʹߩݙͰ͖Δਓࡐ΋ଟ͍͘Δ ҟͳΔઐ໳ੑΛ࣋ͬͨϝϯόʔ͕ڠྗͰ͖Δ͘͠ΈΛ࡞Γɺ ͦΕΛ׆͔ͯ͠αʔϏεΛΑΓྑ͍͖͍ͯͨ͘͠
76. ࠓ೔఻͍͑ͨ͜ͱ3 σʔλʹؔΘΔ࢓ࣄʹٻΊΒΕΔ͜ͱ͸ʁ
77. σʔλ׆༻ͱઐ໳ੑ σʔλʹؔΘΔઐ໳෼໺͸ඇৗʹͨ͘͞Μ͋Δ • ΞφϦετ, σʔλαΠΤϯςΟετ, Ϛʔέλʔ, ... • ػցֶशΤϯδχΞ, σʔλΤϯδχΞ, ... ʮ͜Ε͕Ͱ͖Ε͹શ෦Ͱ͖Δʯͱ͍ͬͨۜͷ஄ؙ͸ଘࡏ͠ͳ͍ • PythonͰػցֶश͚ͩͰ͖Ε͹OK → ❌ • RͰσʔλ෼ੳ͚ͩͰ͖Ε͹OK → ❌
78. Ͳ͏΍ͬͯઐ໳ੑΛ਎ʹ͚ͭΔʁ ͸ͨΒ͘ਓશһ͕৘ใܥग़਎Ͱ΋ͳ͍͠ɺ࠷ॳ͔ΒWebʹৄ͍͠Θ͚Ͱ΋ͳ͍ ͱ͸͍͑ɺ୭Ͱ΋ԿΒ͔ͷσʔλ΍ϩδοΫʹ৮ΕΔػձ͸ҙ֎ͱ͋Δ • ੜ෺ܥͷݚڀࣨͰɺ࣮ݧσʔλͷ౷ܭݕఆΛ͍ͯͨ͠ • લ৬͕ӦۀͰɺച্਺஋Λ΋ͱʹࢪࡦΛཱ͍ͯͯͨ • εϙʔπ׆ಈͰɺ૬खνʔϜͷઓྗ෼ੳΛ͍ͯͨ͠ → ԿΒ͔ͷ͖͔͚͔ͬΒσʔλʹڵຯΛ࣋ͭ
79. σʔλͱ͏·͘෇͖߹͏ʹ͸ʁ τϨϯυʹ૸Βͣɺ·ͣ͸ࣗ෼ͷ໨తͷͨΊʹσʔλΛͲ͏࢖͑Δ͔Λߟ͑Δ ͦͷ্Ͱɺࣗ෼ʹͰ͖ΔखஈΛߟ͑ͯࢼߦࡨޡ͢Δ • ࠓ͋Δσʔλ͔ΒɺͲΜͳ݁Ռ͕ಘΒΕΕ͹໨త͕ୡ੒Ͱ͖Δʁ • ͦͷͨΊʹ͸σʔλʹͲΜͳख๏΍ཧ࿦Λద༻͢Δʁ • ࠓ͋Δσʔλ͕ෆे෼ͳΒɺͲ͏΍Ε͹ิ͏͜ͱ͕Ͱ͖Δʁ ࠷ॳ͔Β͏·͍͘͘͜ͱ͸গͳ͍͕ɺ্ख͍͔͘ͳ͍ཧ༝Λߟ͑Δͷ͕େࣄ
80. σʔλʹؔΘΔ࢓ࣄʹͭ͘ʹ͸ʁ ࣮ࡍͷ࢓ࣄͰσʔλ͕Ͳ͏࢖ΘΕΔ͔஌Γ͍ͨ → Πϯλʔϯ΍ΞϧόΠτΛ׆༻͢Δ ΠϯλʔϯɾΞϧόΠτબͼͷϙΠϯτ • ֶੜ͚ͩͷϋοΧιϯͰ͸ͳ͘ɺ࣮຿ʹؔΘΕΔػձΛݟ͚ͭΔ • ʮ͜Ε͔Β΍͍͖͍ͬͯͨʯͰ͸ͳ͘ʮ͢Ͱʹ΍͍ͬͯΔʯձࣾΛݟ͚ͭΔ • σʔλʹؔΘΔͳΒɺαϙʔτ͢Δ࢓૊Έ͕੔͍ͬͯΔ͔Λฉ͍ͯΈΔ
81. ͍͞͝ʹ
82. ࠓ೔ͷ࿩ʹڵຯΛ࣋ͬͯ΋Β͑ͨΒ…
83. αϚʔΠϯλʔϯ • ظؒ : Ԡ૬ஊ • ର৅ : ػցֶशɾσʔλ෼ੳɾαʔόαΠυ։ൃͷܦݧऀ • څ༩ : ࣌څ1200ԁʙ (ަ௨අࢧڅɾԕํͷ৔߹͸॓ധࢪઃख഑) • ಺༰ྫ : • ΫνίϛΛར༻ͨࣗ͠વݴޠॲཧʹΑΔ৽ػೳͷ։ൃ • Factorization Machine ͳͲͷΞϧΰϦζϜΛར༻ͨ͠Ϩίϝϯυ։ൃ • GCP্ͰͷABςετɾόϯσΟοτج൫ͷ։ൃ
84. Q & A