Upgrade to Pro — share decks privately, control downloads, hide ads and more …

당근마켓이 데이터와 함께 노는 법 - 2020 데이터야놀자

F52763df3c2cfd37922efdbe957aaf23?s=47 Bumsoo Kim
October 17, 2020

당근마켓이 데이터와 함께 노는 법 - 2020 데이터야놀자

2020 데이터야놀자 https://datayanolja.github.io/
당근마켓이 데이터와 함께 노는 법 - 데이터로 회사도 지키고 커리어도 지키는 재미
2020.10.17 Sat

F52763df3c2cfd37922efdbe957aaf23?s=128

Bumsoo Kim

October 17, 2020
Tweet

Transcript

  1. ׼Ӕ݃௄੉ ؘ੉ఠ৬ ೣԋ ֢ח ߨ ੉ਔਗ / ׼Ӕ݃௄ (ؘ੉ఠ ࢎ੉঱౭झ౟)

    
 ӣߧࣻ / ׼Ӕ݃௄ (ؘ੉ఠ ূ૑פয)
  2. ӣߧࣻ - ׼Ӕ݃௄ ؘ੉ఠ ূ૑פয - (੹) ஠೒ۖ(ോݓझݽ࠽ܻ౭) ؘ੉ఠࣄ ܻ٘

    - (੹) ߬੉Ӗ௏٘ ؘ੉ఠ ূ૑פয݂ ݒפ੷ 01 / 42 ߊ಴੗ ࣗѐ https://bsk.im @bskim45 ֎Ҋ৴ ‘׼Ӕ݃௄' ಞী ١੢೮؍ Ӓ ࢎۈ
  3. ੉ਔਗ - ׼Ӕ݃௄ ؘ੉ఠ ࢎ੉঱౭झ౟ - (੹) Naver Clova Research

    ੋఢ - (੹) ઁ੉݃೒ ݠन۞׬ ূ૑פয 02 / 42 ߊ಴੗ ࣗѐ @dnddnjs
  4. - ׼Ӕ݃௄ ࣗѐ - ֥੉ 1: ೖ٘ ୒ࣗೞӝ - ֥੉

    2: ؘ੉ఠ ҕәೞӝ 03 / 42 ߊ಴ ղਊ
  5. 04 / 42 ׼Ӕ݃௄ ࣗѐ

  6. 05 / 42 ׼Ӕ݃௄ ࣗѐ

  7. 06 / 42 ׼Ӕ݃௄ ࣗѐ

  8. ୐ ߣ૩ ֥੉ ೖ٘ ୒ࣗೞӝ 07 / 42

  9. ઺ҊѢې : Ӕ୊੄ ੉ਓҗ ॳ૑ ঋח ޛѤਸ ࣗࣗೞѱ Ѣې →

    য਎ܻ૑ ঋ਷ Ӗ : ੹ޙ ౸ݒ স੗, ૑৉җ ҙ۲ হח স୓ ഘࠁ ١ 08 / 42 ׼Ӕ݃௄੄ ஂ૑
  10. ز֎ࢤഝ : Ӕ୊੄ ੉ਓҗ ز֎ী ҙೠ ׮নೠ ੉ঠӝܳ ೞח Ҕ

    → য਎ܻ૑ ঋ਷ Ӗ : ઺ҊѢې, স୓ ഘࠁ ١ 09 / 42 ׼Ӕ݃௄੄ ஂ૑
  11. ઺Ҋݒޛ, ز֎ ੉ঠӝܳ ݒѐ୓۽ ೠ ૓૞ ز֎ ࢎח ࢎۈٜ੄ োѾ

    10 / 42 ׼Ӕ݃௄੉ ୶ҳೞח ೖ٘ ҃೷
  12. ݅ড ׼Ӕ݃௄җ য਎ܻ૑ ঋח Ӗਸ ݠन۞׬ਵ۽ ঌইյ ࣻ ੓׮ݶ? →

    ࢎਊ੗о ೖ٘ীࢲ ز֎ ੉ਓٜ੄ Ӗ݅ ࠅ ࣻ ੓ѱ ೞӝ 11 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂
  13. 12 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂

    Ӗ ੘ࢿ ೖ٘ ஶబஎ
  14. 13 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂

    Ӗ ੘ࢿ ೖ٘ ஶబஎ
  15. 14 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ݠन۞׬ ࠙ܨ җ੿

    p(೙ఠ݂ ஶబஎ)
  16. 15 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ೖ٘ ੗ز नҊ

    p(೙ఠ݂ ஶబஎ) > 0.9 ࢎਊ੗ ై಴ p(೙ఠ݂ ஶబஎ) <= 0.5 0.5 < p(೙ఠ݂ ஶబஎ) <= 0.9 ݠन۞׬ ࠙ܨ җ੿
  17. 16 / 42 ݠन۞׬ ࠙ܨ җ੿ যڃ ੿ࠁܳ ഝਊ೧ࢲ Ӗਸ

    ࠙ܨ೧ঠ ೡө? → ׼Ӕ݃௄੄ ׮নೠ ഋక੄ ѱदӖਸ ࢓ಝࠁӝ
  18. 17 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ:

    ղਊী ೧׼ೞח ఫझ౟, ࢎਊ੗ ץ֎੐
  19. 18 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ:

    ղਊী ೧׼ೞח ఫझ౟ ݠन۞׬ ݽ؛ ࠙ܨ
  20. 19 / 42 ؘ੉ఠ ળ࠺ ೟णী ࢎਊೠ ؘ੉ఠ ࣻ -

    ز֎ࢤഝ ѱदӖ ؘ੉ఠ : 4݅ѐ, 10ѐ ஠పҊܻ - ઺ҊѢې ѱदӖ ؘ੉ఠ : 40݅ѐ, 11ѐ ஠పҊܻ
  21. 20 / 42 ݠन۞׬ ݽ؛ ೟ण ࢎۈ੉ ࠙ܨೡ ٸ :

    ఫझ౟ܳ ੍Ҋ ੉೧೧ࢲ ࠙ܨ ݠन۞׬੉ ࠙ܨೡ ٸ : ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ ష௼ա੉૚ ݠन۞׬ ݽ؛
  22. 21 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺

    ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ → sentencepiece ۄ੉࠳۞ܻ ࢎਊ ['▁࣠', 'ب', 'ীࢲ', '▁যܽ੉', '▁੉', 'ࡈ', '▁ҕр', 'ਬ', ' ૑', '੢஖', '▁೧', 'ࠁ', 'न', '▁࠙', '▁੓ਸө', 'ਃ', '??', ' ▁ࠗ', 'ୌ', '▁઺', 'ز', 'ਵ۽', '▁஖', 'җ', '▁׮פ', '׮', ' о', '▁࣠', 'ب۽', '▁੉', 'ࢎ৬', 'ࢲ', '▁஖', 'җ', 'ܳ', '▁ ৤ѹ', 'ঠ', 'غ', '֎ਃ', '.', '.', '▁', '഑', 'द', '▁୶ୌ', '೧', ' ઱', 'प', '݅', 'ೠ', '▁஖', 'җ', 'о', '▁੓ਸө', 'ਃ', '?'] [2, 2869, 5859, 6903, 3234, 3647, 6461, 1024, 7063, 7318, 7189, 4998, 6364, 6733, 2468, 3881, 6999, 260, 2423, 7422, 4257, 5872, 7078, 4617, 5468, 1564, 5782, 5330, 2869, 5860, 3647, 6507, 6553, 4617, 5468, 6116, 3452, 6844, 5886, 5703, 54, 54, 517, 7930, 6705, 4559, 7848, 7276, 6738, 6150, 7828, 4617, 5468, 5330, 3881, 6999, 258, 3]
  23. 22 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛

    : BERT
  24. 23 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛

    : BERT ۄ݂߰ೠ ؘ੉ఠ ࠙ܨ ೟ण ׼Ӕ݃௄ ੹୓ ѱ दӖ ؘ੉ఠ ࢎ੹ ೟ण
  25. 24 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ೟ण → huggingface੄

  26. 25 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ
  27. 26 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ - ೧Ѿ଼ 1 : loss function੄ ߸ഋ (focal loss, etc..) - ೧Ѿ଼ 2 : ؘ੉ఠ۝੉ ੸਷ ௿ېझܳ ೟णೡ ٸ ؊ ੗઱ ࢎਊ ੉޷૑ ୹୊ : https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets
  28. 27 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण

    : f1 score 0.84
  29. 28 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? ؘ੉ఠ

    ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ
  30. 29 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ torch-serve
  31. 30 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve
  32. 31 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? →

    model serverܳ ٜ݅যঠ ೣ → torch-serve → flask server ࠁ׮ э਷ दр زউ 5ߓ ؊ ݆਷ ਃ୒ਸ ୊ܻ оמ → ഻ঁ ؊ ੸਷ cpu, memory ࢎਊ
  33. 32 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ਸ ഝਊ೧ࢲ ૓૞ ੉ਓٜਸ োѾ

    - ࢲ࠺झ੄ ஂ૑ܳ ੉೧ - ഝਊೡ ੿ࠁ੄ ഋక ੿ೞӝ - ؘ੉ఠࣇ ݽਵӝ - ݽ؛ ೟णೞӝ - ݽ؛ ࢲࡂೞӝ → ੤޷!
  34. ف ߣ૩ ֥੉ ؘ੉ఠ ҕәೞӝ 33 / 42

  35. ࢲ࠺झо ࢿ੢ೞݶ ഥࢎب ࢿ੢ೞҊ, ؘ੉ఠب ࢿ੢೧ਃ ؘ੉ఠ੄ ߧਤ৬ ௼ӝо טযաਃ

    ࡅܰҊ উ੹ೞѱ ੷੢ೞӝ ؘ੉ఠܳ ॳח ࢎۈب ݆ইઉਃ औҊ ࡅܰѱ ԁղॳӝ MAU 1000݅੄ ޖѱ 34 / 42
  36. ݆਷ ؘ੉ఠܳ “־ҳա औҊ ࡅܰҊ উ੹ೞѱ” 35 / 42 ־ҳա

    औҊ ֫਷ ੽Ӕࢿ = ؘ੉ఠ ޹઱ച(Democratization) ࡅܰѱ ࠻ؘ੉ఠী ؀೧ࢲب աࢁ૑ ঋ਷ ࣘب۽ উ੹ೞѱ ࠁউ / ੽Ӕઁয, ؀ਊ۝ / ഛ੢ࢿ
  37. ELT, not ETL 36 / 42 Extract-Load-Transform vs Extract-Transform-Load •ؘ੉ఠ੄

    ੷੢/୊ܻ ࠺ਊ੉ ऱҊ ࡈۄ઎যਃ (feat. ௿ۄ਋٘) •ؘ੉ఠܳ ৘ࢁѱ ऺח Ѫ ࠁ׮, ੌױ ऺҊ ੜ ഝਊೞח Ѫ੉ ؊ ઺ਃೞ׮Ҋ ޺যਃ
  38. Lambda Architecture •S3 + Athena •S3 + Spark (EMR) Ҋয়ә

    ࠙ࢳ •S3 + Jupyter •S3 + Spark + Zeppelin/Jupyter 37 / 42 Source: https://dzone.com/articles/lambda-architecture-with-apache-spark
  39. ׼Ӕ݃௄ ݫੋ ؘ੉ఠ ਝযೞ਋झ •֫਷ ੽Ӕࢿ: ಞܻೠ ਢ ௑ࣛ UI

    •ഛ੢ࢿ: ࢚؀੸ਵ۽ ੷۴ೠ ࠺ਊ, ࡅܲ ઑഥ •ࠁউ: GCP ӝࠄ ઁҕ, ੽Ӕઁয(ACL) ಞܻ ਊبী ٮۄ झ౟ܻ߁ ప੉࠶, External ప੉࠶ ࢎਊ ਋ܻ੄ ޷ې: BigQuery Omni 38 / 42 Google BigQuery
  40. •S3 + Athena •S3 + Spark (EMR / Glue) Ҋয়ә

    ࠙ࢳ •S3 + Spark + Zeppelin/Jupyter 39 / 42 ೞ٘ೞѱ ֥Ҋ रਸ ٸ
  41. Firebase Analytics੄ ৈ۞о૑ ױ੼ٜ ؘ੉ఠܳ ਋ܻо ਗೞח ؀۽ ஶ౟܀ ೡ

    ࣻо হযਃ •߈ա੺ ੉࢚ ןѱ ٜযয়ח ؘ੉ఠٜ ؘ੉ఠ ನݘਸ ߄Ե ࣻ হযਃ •ߧਊ ೒ۖಬ੄ ౠࢿ࢚ ೙ਃহҊ ࠂ੟ೠ ೙ٜ٘ •ؘ੉ఠ झப ࠺ਊ/௪ܻ दр ૐо 40 / 42 Self-BI(Business Intelligence) दझమ
  42. ਗࠄ ప੉࠶(੉߮౟, झշࢫ) → оҕػ ࠙ࢳਊ ప੉࠶ → दпച, ࢲ࠺झ

    ١ਸ ਤೠ ప੉࠶ 41 / 42 Source: Gaming on AWS 2018: ؘ࠳दझఠૉ ؘ੉ఠ ۨ੉௼ ҳ୷ ੉ঠӝ ױ҅੸ ౭য(Tier) ప੉࠶
  43. 42 / 42 https://dngn.kr/join-us-0