당근마켓이 데이터와 함께 노는 법 - 2020 데이터야놀자

2020 데이터야놀자 https://datayanolja.github.io/
당근마켓이 데이터와 함께 노는 법 - 데이터로 회사도 지키고 커리어도 지키는 재미
2020.10.17 Sat

Transcript

1. ׼Ӕ݃௄੉ ؘ੉ఠ৬
   
   
   ೣԋ ֢ח ߨ
   ੉ਔਗ / ׼Ӕ݃௄ (ؘ੉ఠ ࢎ੉঱౭झ౟)
   

   ӣߧࣻ / ׼Ӕ݃௄ (ؘ੉ఠ ূ૑פয)
   

   View Slide

2. ӣߧࣻ
   
   
   - ׼Ӕ݃௄ ؘ੉ఠ ূ૑פয
   
   
   - (੹) ஠೒ۖ(ോݓझݽ࠽ܻ౭) ؘ੉ఠࣄ ܻ٘
   
   
   - (੹) ߬੉Ӗ௏٘ ؘ੉ఠ ূ૑פয݂ ݒפ੷
   01 / 42
   ߊ಴੗ ࣗѐ
   https://bsk.im
   @bskim45
   ֎Ҋ৴ ‘׼Ӕ݃௄' ಞী ١੢೮؍ Ӓ ࢎۈ
   

   View Slide

3. ੉ਔਗ
   
   
   - ׼Ӕ݃௄ ؘ੉ఠ ࢎ੉঱౭झ౟
   
   
   - (੹) Naver Clova Research ੋఢ
   
   
   - (੹) ઁ੉݃೒ ݠन۞׬ ূ૑פয
   02 / 42
   ߊ಴੗ ࣗѐ
   @dnddnjs
   

   View Slide

4. - ׼Ӕ݃௄ ࣗѐ
   
   
   - ֥੉ 1: ೖ٘ ୒ࣗೞӝ
   
   
   - ֥੉ 2: ؘ੉ఠ ҕәೞӝ
   03 / 42
   
   
   ߊ಴ ղਊ
   

   View Slide

5. 04 / 42
   ׼Ӕ݃௄ ࣗѐ
   

   View Slide

6. 05 / 42
   ׼Ӕ݃௄ ࣗѐ
   

   View Slide

7. 06 / 42
   ׼Ӕ݃௄ ࣗѐ
   

   View Slide

8. ୐ ߣ૩ ֥੉
   
   
   ೖ٘ ୒ࣗೞӝ
   
   
   07 / 42
   
   
   

   View Slide

9. ઺ҊѢې : Ӕ୊੄ ੉ਓҗ ॳ૑ ঋח ޛѤਸ ࣗࣗೞѱ Ѣې
   
   
   → য਎ܻ૑ ঋ਷ Ӗ : ੹ޙ ౸ݒ স੗, ૑৉җ ҙ۲ হח স୓ ഘࠁ ١
   08 / 42
   ׼Ӕ݃௄੄ ஂ૑
   

   View Slide

10. ز֎ࢤഝ : Ӕ୊੄ ੉ਓҗ ز֎ী ҙೠ ׮নೠ ੉ঠӝܳ ೞח Ҕ
    
    
    → য਎ܻ૑ ঋ਷ Ӗ : ઺ҊѢې, স୓ ഘࠁ ١
    09 / 42
    ׼Ӕ݃௄੄ ஂ૑
    

    View Slide

11. ઺Ҋݒޛ, ز֎ ੉ঠӝܳ ݒѐ୓۽ ೠ ૓૞ ز֎ ࢎח ࢎۈٜ੄ োѾ
    10 / 42
    ׼Ӕ݃௄੉ ୶ҳೞח ೖ٘ ҃೷
    

    View Slide

12. ݅ড ׼Ӕ݃௄җ য਎ܻ૑ ঋח Ӗਸ ݠन۞׬ਵ۽ ঌইյ ࣻ ੓׮ݶ?
    
    
    → ࢎਊ੗о ೖ٘ীࢲ ز֎ ੉ਓٜ੄ Ӗ݅ ࠅ ࣻ ੓ѱ ೞӝ
    11 / 42
    ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂
    

    View Slide

13. 12 / 42
    ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂
    ݠन۞׬ ݽ؛
    
    
    ೙ఠ݂
    Ӗ ੘ࢿ ೖ٘ ஶబஎ
    

    View Slide

14. 13 / 42
    ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂
    ݠन۞׬ ݽ؛
    
    
    ೙ఠ݂
    Ӗ ੘ࢿ ೖ٘ ஶబஎ
    

    View Slide

15. 14 / 42
    Ӗ ੘ࢿ ݠन۞׬ ݽ؛
    ݠन۞׬ ࠙ܨ җ੿
    p(೙ఠ݂ ஶబஎ)
    

    View Slide

16. 15 / 42
    Ӗ ੘ࢿ ݠन۞׬ ݽ؛
    ೖ٘
    ੗ز नҊ
    p(೙ఠ݂ ஶబஎ) > 0.9
    ࢎਊ੗ ై಴
    p(೙ఠ݂ ஶబஎ) <= 0.5
    0.5 < p(೙ఠ݂ ஶబஎ) <= 0.9
    ݠन۞׬ ࠙ܨ җ੿
    

    View Slide

17. 16 / 42
    ݠन۞׬ ࠙ܨ җ੿
    যڃ ੿ࠁܳ ഝਊ೧ࢲ Ӗਸ ࠙ܨ೧ঠ ೡө?
    
    
    → ׼Ӕ݃௄੄ ׮নೠ ഋక੄ ѱदӖਸ ࢓ಝࠁӝ
    

    View Slide

18. 17 / 42
    ݠन۞׬ ࠙ܨ җ੿
    ѱदӖী ٜয੓ח ҕా ੿ࠁ: ղਊী ೧׼ೞח ఫझ౟, ࢎਊ੗ ץ֎੐
    

    View Slide

19. 18 / 42
    ݠन۞׬ ࠙ܨ җ੿
    ѱदӖী ٜয੓ח ҕా ੿ࠁ: ղਊী ೧׼ೞח ఫझ౟
    ݠन۞׬ ݽ؛ ࠙ܨ
    

    View Slide

20. 19 / 42
    ؘ੉ఠ ળ࠺
    ೟णী ࢎਊೠ ؘ੉ఠ ࣻ
    
    
    - ز֎ࢤഝ ѱदӖ ؘ੉ఠ : 4݅ѐ, 10ѐ ஠పҊܻ
    
    
    - ઺ҊѢې ѱदӖ ؘ੉ఠ : 40݅ѐ, 11ѐ ஠పҊܻ
    

    View Slide

21. 20 / 42
    ݠन۞׬ ݽ؛ ೟ण
    ࢎۈ੉ ࠙ܨೡ ٸ : ఫझ౟ܳ ੍Ҋ ੉೧೧ࢲ ࠙ܨ
    
    
    ݠन۞׬੉ ࠙ܨೡ ٸ : ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ
    ݽ؛ ాҗ
    ష௼ա੉૚ ݠन۞׬ ݽ؛
    

    View Slide

22. 21 / 42
    ݠन۞׬ ݽ؛ ೟ण
    ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ
    
    
    → sentencepiece ۄ੉࠳۞ܻ ࢎਊ
    ['▁࣠', 'ب', 'ীࢲ', '▁যܽ੉', '▁੉', 'ࡈ', '▁ҕр', 'ਬ', '
    ૑', '੢஖', '▁೧', 'ࠁ', 'न', '▁࠙', '▁੓ਸө', 'ਃ', '??', '
    ▁ࠗ', 'ୌ', '▁઺', 'ز', 'ਵ۽', '▁஖', 'җ', '▁׮פ', '׮', '
    о', '▁࣠', 'ب۽', '▁੉', 'ࢎ৬', 'ࢲ', '▁஖', 'җ', 'ܳ', '▁
    ৤ѹ', 'ঠ', 'غ', '֎ਃ', '.', '.', '▁', '഑', 'द', '▁୶ୌ', '೧', '
    ઱', 'प', '݅', 'ೠ', '▁஖', 'җ', 'о', '▁੓ਸө', 'ਃ', '?']
    [2, 2869, 5859, 6903, 3234, 3647,
    6461, 1024, 7063, 7318, 7189, 4998,
    6364, 6733, 2468, 3881, 6999, 260,
    2423, 7422, 4257, 5872, 7078, 4617,
    5468, 1564, 5782, 5330, 2869, 5860,
    3647, 6507, 6553, 4617, 5468, 6116,
    3452, 6844, 5886, 5703, 54, 54, 517,
    7930, 6705, 4559, 7848, 7276, 6738,
    6150, 7828, 4617, 5468, 5330, 3881,
    6999, 258, 3]
    

    View Slide

23. 22 / 42
    ݠन۞׬ ݽ؛ ೟ण
    ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛ : BERT
    

    View Slide

24. 23 / 42
    ݠन۞׬ ݽ؛ ೟ण
    ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛ : BERT
    ۄ݂߰ೠ ؘ੉ఠ ࠙ܨ ೟ण
    ׼Ӕ݃௄ ੹୓ ѱ
    दӖ ؘ੉ఠ
    ࢎ੹ ೟ण
    

    View Slide

25. 24 / 42
    ݠन۞׬ ݽ؛ ೟ण
    BERT ೟ण → huggingface੄
    

    View Slide

26. 25 / 42
    ݠन۞׬ ݽ؛ ೟ण
    BERT ࠙ܨ ݽ؛ ೟ण → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ
    

    View Slide

27. 26 / 42
    ݠन۞׬ ݽ؛ ೟ण
    BERT ࠙ܨ ݽ؛ ೟ण → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ
    
    
    - ೧Ѿ଼ 1 : loss function੄ ߸ഋ (focal loss, etc..)
    
    
    - ೧Ѿ଼ 2 : ؘ੉ఠ۝੉ ੸਷ ௿ېझܳ ೟णೡ ٸ ؊ ੗઱ ࢎਊ
    ੉޷૑ ୹୊ : https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets
    

    View Slide

28. 27 / 42
    ݠन۞׬ ݽ؛ ೟ण
    BERT ࠙ܨ ݽ؛ ೟ण : f1 score 0.84
    

    View Slide

29. 28 / 42
    ݠन۞׬ ੸ਊ
    ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ?
    ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ
    

    View Slide

30. 29 / 42
    ݠन۞׬ ੸ਊ
    ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ?
    
    
    → model serverܳ ٜ݅যঠ ೣ → torch-serve
    ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ
    torch-serve
    

    View Slide

31. 30 / 42
    ݠन۞׬ ੸ਊ
    ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ?
    
    
    → model serverܳ ٜ݅যঠ ೣ → torch-serve
    

    View Slide

32. 31 / 42
    ݠन۞׬ ੸ਊ
    ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ?
    
    
    → model serverܳ ٜ݅যঠ ೣ → torch-serve
    
    
    → flask server ࠁ׮ э਷ दр زউ 5ߓ ؊ ݆਷ ਃ୒ਸ ୊ܻ оמ
    
    
    → ഻ঁ ؊ ੸਷ cpu, memory ࢎਊ
    

    View Slide

33. 32 / 42
    ݠन۞׬ ੸ਊ
    ݠन۞׬ਸ ഝਊ೧ࢲ ૓૞ ੉ਓٜਸ োѾ
    
    
    - ࢲ࠺झ੄ ஂ૑ܳ ੉೧
    
    
    - ഝਊೡ ੿ࠁ੄ ഋక ੿ೞӝ
    
    
    - ؘ੉ఠࣇ ݽਵӝ
    
    
    - ݽ؛ ೟णೞӝ
    
    
    - ݽ؛ ࢲࡂೞӝ
    
    
    → ੤޷!
    

    View Slide

34. ف ߣ૩ ֥੉
    
    
    ؘ੉ఠ ҕәೞӝ
    
    
    33 / 42
    
    
    

    View Slide

35. ࢲ࠺झо ࢿ੢ೞݶ ഥࢎب ࢿ੢ೞҊ, ؘ੉ఠب ࢿ੢೧ਃ
    
    
    ؘ੉ఠ੄ ߧਤ৬ ௼ӝо טযաਃ
    
    
    ࡅܰҊ উ੹ೞѱ ੷੢ೞӝ
    
    
    ؘ੉ఠܳ ॳח ࢎۈب ݆ইઉਃ
    
    
    औҊ ࡅܰѱ ԁղॳӝ
    MAU 1000݅੄ ޖѱ
    34 / 42
    

    View Slide

36. ݆਷ ؘ੉ఠܳ “־ҳա औҊ ࡅܰҊ উ੹ೞѱ”
    35 / 42
    ־ҳա औҊ
    ֫਷ ੽Ӕࢿ
    
    
    = ؘ੉ఠ ޹઱ച(Democratization)
    ࡅܰѱ ࠻ؘ੉ఠী ؀೧ࢲب աࢁ૑ ঋ਷ ࣘب۽
    উ੹ೞѱ ࠁউ / ੽Ӕઁয, ؀ਊ۝ / ഛ੢ࢿ
    

    View Slide

37. ELT, not ETL
    36 / 42
    Extract-Load-Transform vs Extract-Transform-Load
    
    
    ●ؘ੉ఠ੄ ੷੢/୊ܻ ࠺ਊ੉ ऱҊ ࡈۄ઎যਃ (feat. ௿ۄ਋٘)
    
    
    ●ؘ੉ఠܳ ৘ࢁѱ ऺח Ѫ ࠁ׮, ੌױ ऺҊ ੜ ഝਊೞח Ѫ੉
    
    
    ؊ ઺ਃೞ׮Ҋ ޺যਃ
    

    View Slide

38. Lambda Architecture
    ●S3 + Athena
    
    
    ●S3 + Spark (EMR)
    
    
    Ҋয়ә ࠙ࢳ
    
    
    ●S3 + Jupyter
    
    
    ●S3 + Spark + Zeppelin/Jupyter
    37 / 42
    Source: https://dzone.com/articles/lambda-architecture-with-apache-spark
    

    View Slide

39. ׼Ӕ݃௄ ݫੋ ؘ੉ఠ ਝযೞ਋झ
    
    
    ●֫਷ ੽Ӕࢿ: ಞܻೠ ਢ ௑ࣛ UI
    
    
    ●ഛ੢ࢿ: ࢚؀੸ਵ۽ ੷۴ೠ ࠺ਊ, ࡅܲ ઑഥ
    
    
    ●ࠁউ: GCP ӝࠄ ઁҕ, ੽Ӕઁয(ACL) ಞܻ
    
    
    ਊبী ٮۄ झ౟ܻ߁ ప੉࠶, External ప੉࠶ ࢎਊ
    
    
    ਋ܻ੄ ޷ې: BigQuery Omni
    38 / 42
    Google BigQuery
    

    View Slide

40. ●S3 + Athena
    
    
    ●S3 + Spark (EMR / Glue)
    
    
    Ҋয়ә ࠙ࢳ
    
    
    ●S3 + Spark + Zeppelin/Jupyter
    39 / 42
    ೞ٘ೞѱ ֥Ҋ रਸ ٸ
    

    View Slide

41. Firebase Analytics੄ ৈ۞о૑ ױ੼ٜ
    
    
    ؘ੉ఠܳ ਋ܻо ਗೞח ؀۽ ஶ౟܀ ೡ ࣻо হযਃ
    
    
    ●߈ա੺ ੉࢚ ןѱ ٜযয়ח ؘ੉ఠٜ
    
    
    ؘ੉ఠ ನݘਸ ߄Ե ࣻ হযਃ
    
    
    ●ߧਊ ೒ۖಬ੄ ౠࢿ࢚ ೙ਃহҊ ࠂ੟ೠ ೙ٜ٘
    
    
    ●ؘ੉ఠ झப ࠺ਊ/௪ܻ दр ૐо
    40 / 42
    Self-BI(Business Intelligence) दझమ
    

    View Slide

42. ਗࠄ ప੉࠶(੉߮౟, झշࢫ) → оҕػ ࠙ࢳਊ ప੉࠶
    
    
    → दпച, ࢲ࠺झ ١ਸ ਤೠ ప੉࠶
    41 / 42
    Source: Gaming on AWS 2018: ؘ࠳दझఠૉ ؘ੉ఠ ۨ੉௼ ҳ୷ ੉ঠӝ
    ױ҅੸ ౭য(Tier) ప੉࠶
    

    View Slide

43. 42 / 42
    https://dngn.kr/join-us-0
    

    View Slide