당근마켓이 데이터와 함께 노는 법 - 2020 데이터야놀자

2020 데이터야놀자 https://datayanolja.github.io/
당근마켓이 데이터와 함께 노는 법 - 데이터로 회사도 지키고 커리어도 지키는 재미
2020.10.17 Sat

Transcript

1. ׼Ӕ݃௄੉ ؘ੉ఠ৬ ೣԋ ֢ח ߨ ੉ਔਗ / ׼Ӕ݃௄ (ؘ੉ఠ ࢎ੉঱౭झ౟) 
 ӣߧࣻ / ׼Ӕ݃௄ (ؘ੉ఠ ূ૑פয)
2. ӣߧࣻ - ׼Ӕ݃௄ ؘ੉ఠ ূ૑פয - (੹) ஠೒ۖ(ോݓझݽ࠽ܻ౭) ؘ੉ఠࣄ ܻ٘ - (੹) ߬੉Ӗ௏٘ ؘ੉ఠ ূ૑פয݂ ݒפ੷ 01 / 42 ߊ಴੗ ࣗѐ https://bsk.im @bskim45 ֎Ҋ৴ ‘׼Ӕ݃௄' ಞী ١੢೮؍ Ӓ ࢎۈ
3. ੉ਔਗ - ׼Ӕ݃௄ ؘ੉ఠ ࢎ੉঱౭झ౟ - (੹) Naver Clova Research ੋఢ - (੹) ઁ੉݃೒ ݠन۞׬ ূ૑פয 02 / 42 ߊ಴੗ ࣗѐ @dnddnjs
4. - ׼Ӕ݃௄ ࣗѐ - ֥੉ 1: ೖ٘ ୒ࣗೞӝ - ֥੉ 2: ؘ੉ఠ ҕәೞӝ 03 / 42 ߊ಴ ղਊ
5. 04 / 42 ׼Ӕ݃௄ ࣗѐ
6. 05 / 42 ׼Ӕ݃௄ ࣗѐ
7. 06 / 42 ׼Ӕ݃௄ ࣗѐ
8. ୐ ߣ૩ ֥੉ ೖ٘ ୒ࣗೞӝ 07 / 42
9. ઺ҊѢې : Ӕ୊੄ ੉ਓҗ ॳ૑ ঋח ޛѤਸ ࣗࣗೞѱ Ѣې → য਎ܻ૑ ঋ਷ Ӗ : ੹ޙ ౸ݒ স੗, ૑৉җ ҙ۲ হח স୓ ഘࠁ ١ 08 / 42 ׼Ӕ݃௄੄ ஂ૑
10. ز֎ࢤഝ : Ӕ୊੄ ੉ਓҗ ز֎ী ҙೠ ׮নೠ ੉ঠӝܳ ೞח Ҕ → য਎ܻ૑ ঋ਷ Ӗ : ઺ҊѢې, স୓ ഘࠁ ١ 09 / 42 ׼Ӕ݃௄੄ ஂ૑
11. ઺Ҋݒޛ, ز֎ ੉ঠӝܳ ݒѐ୓۽ ೠ ૓૞ ز֎ ࢎח ࢎۈٜ੄ োѾ 10 / 42 ׼Ӕ݃௄੉ ୶ҳೞח ೖ٘ ҃೷
12. ݅ড ׼Ӕ݃௄җ য਎ܻ૑ ঋח Ӗਸ ݠन۞׬ਵ۽ ঌইյ ࣻ ੓׮ݶ? → ࢎਊ੗о ೖ٘ীࢲ ز֎ ੉ਓٜ੄ Ӗ݅ ࠅ ࣻ ੓ѱ ೞӝ 11 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂
13. 12 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂ Ӗ ੘ࢿ ೖ٘ ஶబஎ
14. 13 / 42 ݠन۞׬ਸ ഝਊೠ ೖ٘ ೙ఠ݂ ݠन۞׬ ݽ؛ ೙ఠ݂ Ӗ ੘ࢿ ೖ٘ ஶబஎ
15. 14 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ݠन۞׬ ࠙ܨ җ੿ p(೙ఠ݂ ஶబஎ)
16. 15 / 42 Ӗ ੘ࢿ ݠन۞׬ ݽ؛ ೖ٘ ੗ز नҊ p(೙ఠ݂ ஶబஎ) > 0.9 ࢎਊ੗ ై಴ p(೙ఠ݂ ஶబஎ) <= 0.5 0.5 < p(೙ఠ݂ ஶబஎ) <= 0.9 ݠन۞׬ ࠙ܨ җ੿
17. 16 / 42 ݠन۞׬ ࠙ܨ җ੿ যڃ ੿ࠁܳ ഝਊ೧ࢲ Ӗਸ ࠙ܨ೧ঠ ೡө? → ׼Ӕ݃௄੄ ׮নೠ ഋక੄ ѱदӖਸ ࢓ಝࠁӝ
18. 17 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ: ղਊী ೧׼ೞח ఫझ౟, ࢎਊ੗ ץ֎੐
19. 18 / 42 ݠन۞׬ ࠙ܨ җ੿ ѱदӖী ٜয੓ח ҕా ੿ࠁ: ղਊী ೧׼ೞח ఫझ౟ ݠन۞׬ ݽ؛ ࠙ܨ
20. 19 / 42 ؘ੉ఠ ળ࠺ ೟णী ࢎਊೠ ؘ੉ఠ ࣻ - ز֎ࢤഝ ѱदӖ ؘ੉ఠ : 4݅ѐ, 10ѐ ஠పҊܻ - ઺ҊѢې ѱदӖ ؘ੉ఠ : 40݅ѐ, 11ѐ ஠పҊܻ
21. 20 / 42 ݠन۞׬ ݽ؛ ೟ण ࢎۈ੉ ࠙ܨೡ ٸ : ఫझ౟ܳ ੍Ҋ ੉೧೧ࢲ ࠙ܨ ݠन۞׬੉ ࠙ܨೡ ٸ : ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ ష௼ա੉૚ ݠन۞׬ ݽ؛
22. 21 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ܳ “ష௾”ਵ۽ ଂђ ׮਺ ं੗۽ ߸ജ೧ࢲ ݽ؛ ాҗ → sentencepiece ۄ੉࠳۞ܻ ࢎਊ ['▁࣠', 'ب', 'ীࢲ', '▁যܽ੉', '▁੉', 'ࡈ', '▁ҕр', 'ਬ', ' ૑', '੢஖', '▁೧', 'ࠁ', 'न', '▁࠙', '▁੓ਸө', 'ਃ', '??', ' ▁ࠗ', 'ୌ', '▁઺', 'ز', 'ਵ۽', '▁஖', 'җ', '▁׮פ', '׮', ' о', '▁࣠', 'ب۽', '▁੉', 'ࢎ৬', 'ࢲ', '▁஖', 'җ', 'ܳ', '▁ ৤ѹ', 'ঠ', 'غ', '֎ਃ', '.', '.', '▁', '഑', 'द', '▁୶ୌ', '೧', ' ઱', 'प', '݅', 'ೠ', '▁஖', 'җ', 'о', '▁੓ਸө', 'ਃ', '?'] [2, 2869, 5859, 6903, 3234, 3647, 6461, 1024, 7063, 7318, 7189, 4998, 6364, 6733, 2468, 3881, 6999, 260, 2423, 7422, 4257, 5872, 7078, 4617, 5468, 1564, 5782, 5330, 2869, 5860, 3647, 6507, 6553, 4617, 5468, 6116, 3452, 6844, 5886, 5703, 54, 54, 517, 7930, 6705, 4559, 7848, 7276, 6738, 6150, 7828, 4617, 5468, 5330, 3881, 6999, 258, 3]
23. 22 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛ : BERT
24. 23 / 42 ݠन۞׬ ݽ؛ ೟ण ఫझ౟ ࠙ܨীࢲ ӝࠄ ݽ؛ : BERT ۄ݂߰ೠ ؘ੉ఠ ࠙ܨ ೟ण ׼Ӕ݃௄ ੹୓ ѱ दӖ ؘ੉ఠ ࢎ੹ ೟ण
25. 24 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ೟ण → huggingface੄
26. 25 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ
27. 26 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण → ޙઁ : ௿ېझ ߹۽ ࠛӐഋ੉ बೣ - ೧Ѿ଼ 1 : loss function੄ ߸ഋ (focal loss, etc..) - ೧Ѿ଼ 2 : ؘ੉ఠ۝੉ ੸਷ ௿ېझܳ ೟णೡ ٸ ؊ ੗઱ ࢎਊ ੉޷૑ ୹୊ : https://www.kaggle.com/rafjaa/resampling-strategies-for-imbalanced-datasets
28. 27 / 42 ݠन۞׬ ݽ؛ ೟ण BERT ࠙ܨ ݽ؛ ೟ण : f1 score 0.84
29. 28 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ
30. 29 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? → model serverܳ ٜ݅যঠ ೣ → torch-serve ؘ੉ఠ ࠙ܨ ೟ण ࢲ࠺झ ੸ਊ torch-serve
31. 30 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? → model serverܳ ٜ݅যঠ ೣ → torch-serve
32. 31 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ ݽ؛ਸ ࢲ࠺झী ੸ਊೞ۰ݶ? → model serverܳ ٜ݅যঠ ೣ → torch-serve → flask server ࠁ׮ э਷ दр زউ 5ߓ ؊ ݆਷ ਃ୒ਸ ୊ܻ оמ → ഻ঁ ؊ ੸਷ cpu, memory ࢎਊ
33. 32 / 42 ݠन۞׬ ੸ਊ ݠन۞׬ਸ ഝਊ೧ࢲ ૓૞ ੉ਓٜਸ োѾ - ࢲ࠺झ੄ ஂ૑ܳ ੉೧ - ഝਊೡ ੿ࠁ੄ ഋక ੿ೞӝ - ؘ੉ఠࣇ ݽਵӝ - ݽ؛ ೟णೞӝ - ݽ؛ ࢲࡂೞӝ → ੤޷!
34. ف ߣ૩ ֥੉ ؘ੉ఠ ҕәೞӝ 33 / 42
35. ࢲ࠺झо ࢿ੢ೞݶ ഥࢎب ࢿ੢ೞҊ, ؘ੉ఠب ࢿ੢೧ਃ ؘ੉ఠ੄ ߧਤ৬ ௼ӝо טযաਃ ࡅܰҊ উ੹ೞѱ ੷੢ೞӝ ؘ੉ఠܳ ॳח ࢎۈب ݆ইઉਃ औҊ ࡅܰѱ ԁղॳӝ MAU 1000݅੄ ޖѱ 34 / 42
36. ݆਷ ؘ੉ఠܳ “־ҳա औҊ ࡅܰҊ উ੹ೞѱ” 35 / 42 ־ҳա औҊ ֫਷ ੽Ӕࢿ = ؘ੉ఠ ޹઱ച(Democratization) ࡅܰѱ ࠻ؘ੉ఠী ؀೧ࢲب աࢁ૑ ঋ਷ ࣘب۽ উ੹ೞѱ ࠁউ / ੽Ӕઁয, ؀ਊ۝ / ഛ੢ࢿ
37. ELT, not ETL 36 / 42 Extract-Load-Transform vs Extract-Transform-Load •ؘ੉ఠ੄ ੷੢/୊ܻ ࠺ਊ੉ ऱҊ ࡈۄ઎যਃ (feat. ௿ۄ਋٘) •ؘ੉ఠܳ ৘ࢁѱ ऺח Ѫ ࠁ׮, ੌױ ऺҊ ੜ ഝਊೞח Ѫ੉ ؊ ઺ਃೞ׮Ҋ ޺যਃ
38. Lambda Architecture •S3 + Athena •S3 + Spark (EMR) Ҋয়ә ࠙ࢳ •S3 + Jupyter •S3 + Spark + Zeppelin/Jupyter 37 / 42 Source: https://dzone.com/articles/lambda-architecture-with-apache-spark
39. ׼Ӕ݃௄ ݫੋ ؘ੉ఠ ਝযೞ਋झ •֫਷ ੽Ӕࢿ: ಞܻೠ ਢ ௑ࣛ UI •ഛ੢ࢿ: ࢚؀੸ਵ۽ ੷۴ೠ ࠺ਊ, ࡅܲ ઑഥ •ࠁউ: GCP ӝࠄ ઁҕ, ੽Ӕઁয(ACL) ಞܻ ਊبী ٮۄ झ౟ܻ߁ ప੉࠶, External ప੉࠶ ࢎਊ ਋ܻ੄ ޷ې: BigQuery Omni 38 / 42 Google BigQuery
40. •S3 + Athena •S3 + Spark (EMR / Glue) Ҋয়ә ࠙ࢳ •S3 + Spark + Zeppelin/Jupyter 39 / 42 ೞ٘ೞѱ ֥Ҋ रਸ ٸ
41. Firebase Analytics੄ ৈ۞о૑ ױ੼ٜ ؘ੉ఠܳ ਋ܻо ਗೞח ؀۽ ஶ౟܀ ೡ ࣻо হযਃ •߈ա੺ ੉࢚ ןѱ ٜযয়ח ؘ੉ఠٜ ؘ੉ఠ ನݘਸ ߄Ե ࣻ হযਃ •ߧਊ ೒ۖಬ੄ ౠࢿ࢚ ೙ਃহҊ ࠂ੟ೠ ೙ٜ٘ •ؘ੉ఠ झப ࠺ਊ/௪ܻ दр ૐо 40 / 42 Self-BI(Business Intelligence) दझమ
42. ਗࠄ ప੉࠶(੉߮౟, झշࢫ) → оҕػ ࠙ࢳਊ ప੉࠶ → दпച, ࢲ࠺झ ١ਸ ਤೠ ప੉࠶ 41 / 42 Source: Gaming on AWS 2018: ؘ࠳दझఠૉ ؘ੉ఠ ۨ੉௼ ҳ୷ ੉ঠӝ ױ҅੸ ౭য(Tier) ప੉࠶
43. 42 / 42 https://dngn.kr/join-us-0