당근마켓이 데이터와 함께 노는 법 - 2020 데이터야놀자

Transcript

1. ׼Ӕ݃௄੉ ؘ੉ఠ৬ ೣԋ ֢ח ߨ ੉ਔਗ / ׼Ӕ݃௄ (ؘ੉ఠ ࢎ੉঱౭झ౟)

   
 ӣߧࣻ / ׼Ӕ݃௄ (ؘ੉ఠ ূ૑פয)

2. ӣߧࣻ - ׼Ӕ݃௄ ؘ੉ఠ ূ૑פয - (੹) ஠೒ۖ(ോݓझݽ࠽ܻ౭) ؘ੉ఠࣄ ܻ٘

   - (੹) ߬੉Ӗ௏٘ ؘ੉ఠ ূ૑פয݂ ݒפ੷ 01 / 42 ߊ಴੗ ࣗѐ https://bsk.im @bskim45 ֎Ҋ৴ ‘׼Ӕ݃௄' ಞী ١੢೮؍ Ӓ ࢎۈ

3. ੉ਔਗ - ׼Ӕ݃௄ ؘ੉ఠ ࢎ੉঱౭झ౟ - (੹) Naver Clova Research

   ੋఢ - (੹) ઁ੉݃೒ ݠन۞׬ ূ૑פয 02 / 42 ߊ಴੗ ࣗѐ @dnddnjs

4. - ׼Ӕ݃௄ ࣗѐ - ֥੉ 1: ೖ٘ ୒ࣗೞӝ - ֥੉

   2: ؘ੉ఠ ҕәೞӝ 03 / 42 ߊ಴ ղਊ

5. 04 / 42 ׼Ӕ݃௄ ࣗѐ

6. 05 / 42 ׼Ӕ݃௄ ࣗѐ

7. 06 / 42 ׼Ӕ݃௄ ࣗѐ

8. ୐ ߣ૩ ֥੉ ೖ٘ ୒ࣗೞӝ 07 / 42

9. ઺ҊѢې : Ӕ୊੄ ੉ਓҗ ॳ૑ ঋח ޛѤਸ ࣗࣗೞѱ Ѣې →

   য਎ܻ૑ ঋ਷ Ӗ : ੹ޙ ౸ݒ স੗, ૑৉җ ҙ۲ হח স୓ ഘࠁ ١ 08 / 42 ׼Ӕ݃௄੄ ஂ૑

10. ز֎ࢤഝ : Ӕ୊੄ ੉ਓҗ ز֎ী ҙೠ ׮নೠ ੉ঠӝܳ ೞח Ҕ

    → য਎ܻ૑ ঋ਷ Ӗ : ઺ҊѢې, স୓ ഘࠁ ١ 09 / 42 ׼Ӕ݃௄੄ ஂ૑

11. ઺Ҋݒޛ, ز֎ ੉ঠӝܳ ݒѐ୓۽ ೠ ૓૞ ز֎ ࢎח ࢎۈٜ੄ োѾ

    10 / 42 ׼Ӕ݃௄੉ ୶ҳೞח ೖ٘ ҃೷

12. ݅ড ׼Ӕ݃௄җ য਎ܻ૑ ঋח Ӗਸ ݠन۞׬ਵ۽ ঌইյ ࣻ ੓׮ݶ? →

    ࢎਊ੗о ೖ٘ীࢲ ز֎ ੉ਓٜ੄ Ӗ݅ ࠅ ࣻ ੓ѱ ೞӝ 11 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂

13. 12 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂

    Ӗ ੘ࢿ ೖ٘ ஶబஎ

14. 13 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂

    Ӗ ੘ࢿ ೖ٘ ஶబஎ

15. 14 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ݠन۞׬ ࠙ܨ җ੿

    p(೙ఠ݂ ஶబஎ)

16. 15 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ೖ٘ ੗ز नҊ

    p(೙ఠ݂ ஶబஎ) > 0.9 ࢎਊ੗ ై಴ p(೙ఠ݂ ஶబஎ) <= 0.5 0.5 < p(೙ఠ݂ ஶబஎ) <= 0.9 ݠन۞׬ ࠙ܨ җ੿

17. 16 / 42 ݠन۞׬ ࠙ܨ җ੿ যڃ ੿ࠁܳ ഝਊ೧ࢲ Ӗਸ

    ࠙ܨ೧ঠ ೡө? → ׼Ӕ݃௄੄ ׮নೠ ഋక੄ ѱदӖਸ ࢓ಝࠁӝ

18. 17 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ:

    ղਊী ೧׼ೞח ఫझ౟, ࢎਊ੗ ץ֎੐

19. 18 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ:

    ղਊী ೧׼ೞח ఫझ౟ ݠन۞׬ ݽ؛ ࠙ܨ

20. 19 / 42 ؘ੉ఠ ળ࠺ ೟णী ࢎਊೠ ؘ੉ఠ ࣻ -

    ز֎ࢤഝ ѱदӖ ؘ੉ఠ : 4݅ѐ, 10ѐ ஠పҊܻ - ઺ҊѢې ѱदӖ ؘ੉ఠ : 40݅ѐ, 11ѐ ஠పҊܻ

21. 20 / 42 ݠन۞׬ ݽ؛ ೟ण ࢎۈ੉ ࠙ܨೡ ٸ :

    ఫझ౟ܳ ੍Ҋ ੉೧೧ࢲ ࠙ܨ ݠन۞׬੉ ࠙ܨೡ ٸ : ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ ష௼ա੉૚ ݠन۞׬ ݽ؛

22. 21 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺

    ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ → sentencepiece ۄ੉࠳۞ܻ ࢎਊ ['▁࣠', 'ب', 'ীࢲ', '▁যܽ੉', '▁੉', 'ࡈ', '▁ҕр', 'ਬ', ' ૑', '੢஖', '▁೧', 'ࠁ', 'न', '▁࠙', '▁੓ਸө', 'ਃ', '??', ' ▁ࠗ', 'ୌ', '▁઺', 'ز', 'ਵ۽', '▁஖', 'җ', '▁׮פ', '׮', ' о', '▁࣠', 'ب۽', '▁੉', 'ࢎ৬', 'ࢲ', '▁஖', 'җ', 'ܳ', '▁ ৤ѹ', 'ঠ', 'غ', '֎ਃ', '.', '.', '▁', '഑', 'द', '▁୶ୌ', '೧', ' ઱', 'प', '݅', 'ೠ', '▁஖', 'җ', 'о', '▁੓ਸө', 'ਃ', '?'] [2, 2869, 5859, 6903, 3234, 3647, 6461, 1024, 7063, 7318, 7189, 4998, 6364, 6733, 2468, 3881, 6999, 260, 2423, 7422, 4257, 5872, 7078, 4617, 5468, 1564, 5782, 5330, 2869, 5860, 3647, 6507, 6553, 4617, 5468, 6116, 3452, 6844, 5886, 5703, 54, 54, 517, 7930, 6705, 4559, 7848, 7276, 6738, 6150, 7828, 4617, 5468, 5330, 3881, 6999, 258, 3]

23. 22 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛

    : BERT

24. 23 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛

    : BERT ۄ݂߰ೠ ؘ੉ఠ ࠙ܨ ೟ण ׼Ӕ݃௄ ੹୓ ѱ दӖ ؘ੉ఠ ࢎ੹ ೟ण

25. 24 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ೟ण → huggingface੄

26. 25 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ

27. 26 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ - ೧Ѿ଼ 1 : loss function੄ ߸ഋ (focal loss, etc..) - ೧Ѿ଼ 2 : ؘ੉ఠ۝੉ ੸਷ ௿ېझܳ ೟णೡ ٸ ؊ ੗઱ ࢎਊ ੉޷૑ ୹୊ : https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets

28. 27 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    : f1 score 0.84

29. 28 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? ؘ੉ఠ

    ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ

30. 29 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ torch-serve

31. 30 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve

32. 31 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve → flask server ࠁ׮ э਷ दр زউ 5ߓ ؊ ݆਷ ਃ୒ਸ ୊ܻ оמ → ഻ঁ ؊ ੸਷ cpu, memory ࢎਊ

33. 32 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ਸ ഝਊ೧ࢲ ૓૞ ੉ਓٜਸ োѾ

    - ࢲ࠺झ੄ ஂ૑ܳ ੉೧ - ഝਊೡ ੿ࠁ੄ ഋక ੿ೞӝ - ؘ੉ఠࣇ ݽਵӝ - ݽ؛ ೟णೞӝ - ݽ؛ ࢲࡂೞӝ → ੤޷!

34. ف ߣ૩ ֥੉ ؘ੉ఠ ҕәೞӝ 33 / 42

35. ࢲ࠺झо ࢿ੢ೞݶ ഥࢎب ࢿ੢ೞҊ, ؘ੉ఠب ࢿ੢೧ਃ ؘ੉ఠ੄ ߧਤ৬ ௼ӝо טযաਃ

    ࡅܰҊ উ੹ೞѱ ੷੢ೞӝ ؘ੉ఠܳ ॳח ࢎۈب ݆ইઉਃ औҊ ࡅܰѱ ԁղॳӝ MAU 1000݅੄ ޖѱ 34 / 42

36. ݆਷ ؘ੉ఠܳ “־ҳա औҊ ࡅܰҊ উ੹ೞѱ” 35 / 42 ־ҳա

    औҊ ֫਷ ੽Ӕࢿ = ؘ੉ఠ ޹઱ച(Democratization) ࡅܰѱ ࠻ؘ੉ఠী ؀೧ࢲب աࢁ૑ ঋ਷ ࣘب۽ উ੹ೞѱ ࠁউ / ੽Ӕઁয, ؀ਊ۝ / ഛ੢ࢿ

37. ELT, not ETL 36 / 42 Extract-Load-Transform vs Extract-Transform-Load •ؘ੉ఠ੄

    ੷੢/୊ܻ ࠺ਊ੉ ऱҊ ࡈۄ઎যਃ (feat. ௿ۄ਋٘) •ؘ੉ఠܳ ৘ࢁѱ ऺח Ѫ ࠁ׮, ੌױ ऺҊ ੜ ഝਊೞח Ѫ੉ ؊ ઺ਃೞ׮Ҋ ޺যਃ

38. Lambda Architecture •S3 + Athena •S3 + Spark (EMR) Ҋয়ә

    ࠙ࢳ •S3 + Jupyter •S3 + Spark + Zeppelin/Jupyter 37 / 42 Source: https://dzone.com/articles/lambda-architecture-with-apache-spark

39. ׼Ӕ݃௄ ݫੋ ؘ੉ఠ ਝযೞ਋झ •֫਷ ੽Ӕࢿ: ಞܻೠ ਢ ௑ࣛ UI

    •ഛ੢ࢿ: ࢚؀੸ਵ۽ ੷۴ೠ ࠺ਊ, ࡅܲ ઑഥ •ࠁউ: GCP ӝࠄ ઁҕ, ੽Ӕઁয(ACL) ಞܻ ਊبী ٮۄ झ౟ܻ߁ ప੉࠶, External ప੉࠶ ࢎਊ ਋ܻ੄ ޷ې: BigQuery Omni 38 / 42 Google BigQuery

40. •S3 + Athena •S3 + Spark (EMR / Glue) Ҋয়ә

    ࠙ࢳ •S3 + Spark + Zeppelin/Jupyter 39 / 42 ೞ٘ೞѱ ֥Ҋ रਸ ٸ

41. Firebase Analytics੄ ৈ۞о૑ ױ੼ٜ ؘ੉ఠܳ ਋ܻо ਗೞח ؀۽ ஶ౟܀ ೡ

    ࣻо হযਃ •߈ա੺ ੉࢚ ןѱ ٜযয়ח ؘ੉ఠٜ ؘ੉ఠ ನݘਸ ߄Ե ࣻ হযਃ •ߧਊ ೒ۖಬ੄ ౠࢿ࢚ ೙ਃহҊ ࠂ੟ೠ ೙ٜ٘ •ؘ੉ఠ झப ࠺ਊ/௪ܻ दр ૐо 40 / 42 Self-BI(Business Intelligence) दझమ

42. ਗࠄ ప੉࠶(੉߮౟, झշࢫ) → оҕػ ࠙ࢳਊ ప੉࠶ → दпച, ࢲ࠺झ

    ١ਸ ਤೠ ప੉࠶ 41 / 42 Source: Gaming on AWS 2018: ؘ࠳दझఠૉ ؘ੉ఠ ۨ੉௼ ҳ୷ ੉ঠӝ ױ҅੸ ౭য(Tier) ప੉࠶

43. 42 / 42 https://dngn.kr/join-us-0