파이썬과 커뮤니티와 한국어 오픈데이터

6e474d7c5f627dc8e3eb56ca367968dd?s=47 Lucy Park
August 17, 2019

파이썬과 커뮤니티와 한국어 오픈데이터

PyCon Korea 2019 키노트 발표 자료입니다.

6e474d7c5f627dc8e3eb56ca367968dd?s=128

Lucy Park

August 17, 2019
Tweet

Transcript

  1. 2.

    ߅਷੿ a.k.a. lucypark, echojuliett, e9t • ӝ҅ߣ৉ਸ ೞח ݠन۞׬ ূ૑פয

    • ؘ੉ఠ۽ ૑ध੄ ੢߷ਸ ծ୶Ҋ र਷ ࢎۈ • Yak shaver 2
  2. 3.

    ٸח 2019֙ 5ਘ • ӝഥо ػ׮ݶ ೞҊ र਷ ݈੉ ੓঻য!

    • "ઁউ೧઱࣊ࢲ хࢎ೤פ׮! যڃ ফӝܳ ೧ঠೡ૑ ഛन੉ ٜ૑ח ঋ૑݅ ਬ ੊ೠ ফӝܳ ೡ ࣻ ੓ب۾ ળ࠺೧ࠅѱਃ." • ߊ಴ө૑ 3ѐਘ੉ա ੓ਵפ ળ࠺ೡ दрب ୽࠙ೞ׮! 3
  3. 4.

    ԝԝೠ ҅ദ • ೠҴয NLP੄ য়೑ࣗझ/ؘ੉ఠী ؀ೠ ࢲ߬੉ܳ ೧ࢲ, അടਸ

    ҕਬೞҊ ׮ э੉ ߊ੹दெաо੗Ҋ ઁউ೧ঠѷয! • ః֢౟ۄݶ ੿ࠁܳ ੹׳ೞӝࠁ׮ח ನਊinclusion , ׮নࢿdiversity , ழޭפ౭৬ э਷ о஖ܳ ࠗпೞח ߊ಴ݶ જਸ Ѫ эই • Ӓؘ۠ ژ ౵੉௑੉פө ౵੉ॆী ؀ೠ ফӝب ೧ঠ૑ • Ӓۧ׮ݶ 3֙݅ী konlpyب ખ সؘ੉౟ ೧ࠅө? пઙ ؘ੉ఠࣇਸ konlpy.downloadী ನೣ೧ࢲ ࢎۈٜ੉ औѱ ׮਍۽٘ ߉ѱ ೞݶ જਸ Ѫ э׮! 4
  4. 5.

    ӒܻҊ 2019֙ 8ਘ • konlpyח Ҋசѱ ցޖ ݆׮... э੉ ٜ݅যоਃ!

    • ই૒ ߊ಴੗ܐب হ֎... ƕƕ • ׮೯൤ ੉ܲ ইஜ੄ ୐ ߊ಴פө ࢎۈ੉ ߹۽ উৢѢঠ... 5
  5. 6.

    য়ט ೡ ੉ঠӝ • ޙઁ ੄ध ҕਬ • ழޭפ౭ী ੘ѱա݃

    ӝৈೡ ࣻ ੓঻؍ ѐੋ੸ੋ ҃೷ ҕਬ ঢ়զ ফӝ • ೠҴয য়೑ؘ੉ఠ ੉ঠӝب ઑӘ ઱߸ ࢎۈٜ਷ ݔզ ٜ঻؍ ফӝ *NBHFTPVSDF4BOESBBOE8PP 3JDIBSETHVJEFUPTPGUXBSFEFWFMPQNFOU  6
  6. 8.

    ೐۽Ӓې߁਷ ৵ ೞחо? • ஂ޷ • ݡҊࢎפ્ • ѐੋ੄ ࢿ੢

    • ա੄ ழޭפ౭ী ӝৈೞӝ ੉೦ݾٜ਷࢚ഐߓఋ੸੉૑ঋ׮ 8
  7. 11.

    11

  8. 12.

    12

  9. 19.

    ౱ನಯ http://popong.com • ৘࠺ ٣੗੉ց, ѐߊ੗, ӝദ੗ ١ਵ۽ ҳࢿػ ೟ࢤ

    10ৈ ݺ • ӝࣿ۽ॄ ؀ೠ޹Ҵ ੿஖ܳ ٍൔ٘חѱ ݾ಴ • ૒੽ ࢲ࠺झܳ ٜ݅যࠁݶࢲ ਋ܻо ࢿ੢ೞח Ѫ਷ ؒ! (ژח ؊ ௾ ݾ੸) 1010/(1VCMJD0QFO10MJUJDBMFOHJOFFSJ/( 19
  10. 22.

    ݒ઱ షਃੌ݃׮ ݽৈ ӝദҗ షۿҗ झఠ٣ ؀׮ࣻо ੑߨҗ੿ ؀೧ ইחѱ

    ೞաب হ঻Ѣٚਃ Ӓۧѱ 2ߣ੄ ௼ܻझ݃झܳ ೣԋ ݏ੉ೠ റ... 22
  11. 23.

    ਢࢲ࠺झ "؀ೠ޹Ҵ ੿஖੄ ݽٚ Ѫ" ఍ࢤ ੉Ѫ ٸޙী ௼܀݂ب ৌब൤

    ೞҊ ইऔѱب૑Ә਷ബਊ؀࠺࠺ਊ੉݆੉૑୹ػ׮Ҋ౸ױ೧ࢲ֙ਘࠗఠPVUPGTFSWJDF ೞ૑݅౱ನಯӝо׮दबತࣗࢤ઺੉ۄ׮दࣁ࢚ীաৢࣻب੓যਃ 23
  12. 28.

    Ҵഥ੄ਗ/੄উ ؘ੉ఠ REST API • ࢎप ਢࢲ࠺झח ࣳா੉झ • ੉ѱ

    ਗې ૓೯ೞ۰؍ ೐۽ં౟ • ؊ ݆਷ ࢎۈٜ੉ ੿஖ ؘ੉ఠܳ ࠙ࢳೞѢա ഝਊ೧ࢲ ࢲ࠺झܳ ٜ݅ӝܳ ߄ۗ਺ ইऔѱب૑Ә਷੉ѪبPVUPGTFSWJDFҊ௏٘݅թই੓૑݅ ੉ઁחҴഥীࢲ૒੽ؘ੉ఠܳઁҕ೤פ׮ ഥ੄۾ ੄উ ੄ਗ 28
  13. 30.

    ؀ೠ޹Ҵ ੿஖ܳ ٍൔٜѷ׮Ҋ ࢤпೠ ಁӝ ֈ஖ח ୒ٜ֙਷ • ೠಞਵ۽ח ઱߸੄

    ੘਷ ࢎഥ۽ࠗఠ ੋ੿ਸ ߉ӝب ೮૑݅ • ࣁ࢚਷ Ӓۧѱ औѱ ߸ೞחѱ ইפۄח Ѫਸ ߓਛҊ ࢲߡ ࠺ਊ੉ ࠺ऱ׮חѪب ঌѱ غҊ • ޖ঺ࠁ׮ب ࣗ઺ೠ જ਷ ஘ҳܳ ঳Ҋ 30
  14. 31.

    Ӓ যڃ ҃೷ࠁ׮ ч૓ જ਷ ѐߊ ޙചܳ ߓਛणפ׮ • ޙࢲച

    ೧઱ࣁਃ. ౱ ղ৻৬ ࣗాೞח ߑߨ • Git੄ જ਷ ೐ۑ౭झח ੉Ѣېਃ. ౱ ղ৻৬ ࣗాೡ ٸ ૑ఃݶ જ਷ ݒց • ࢜ ೐۽ં౟ীࢲח Flaskۄח ೐ۨ੐ਕ௼ܳ ॄࠅөਃ? ࢜۽਍ بҳܳ ੊൤ח ߑߨ • ই੉٣যח working prototypeਵ۽ ࠁৈ઱ࣁਃ. ఌ࢚ҕۿਸ ߩযա ౱ਸ ਑૒੉ח ߑߨ • Ӓ ௏٘/ؘ੉ఠ ۄ੉ࣃझо ޤীਃ? ఋੋ੄ ੷੘ޛਸ ઓ઺ೞח ੗ࣁ 31
  15. 36.

    ٸח 2014֙ • ઁ ੹ҕ੉ ਗې ఫझ౟ی਷ ࢚ҙহҊ ं੗ܳ ࠙ࢳೞח

    ؘ੉ఠ݃੉׬ • ೟Ү ೐۽ં౟ܳ ࣻ೯ೞ؍ ઺ ೠҴয ࠙ࢳਸ ಞܻೞѱ ೞӝ ਤ೧ द੘ • ೠҴয ࠙ࢳਸ ೞ۰ݶ ੌױ ష௼ա੉૚ਸ ೧ঠೞחؘ যڌѱ ೞ૑? • ഋకࣗ ࠙ࢳਸ ೞݶ ػ׮חؘ য়೑ࣗझח ੓ա? • ݻ о૑о ੓ח Ѫ э਷ؘ ࢿמ਷ যڌѱ ׮ܰ૑? 36
  16. 37.

    Enter KoNLPy >>> from konlpy.tag import Okt >>> okt =

    Okt() >>> okt.pos('݅աࢲ ߈оਕਃ!') [('݅աࢲ', 'Verb'), ('߈оਕਃ', 'Adjective'), ('!', 'Punctuation')] • пઙ য়೑ࣗझ ഋకࣗ ࠙ࢳӝܳ ݽ਺ • nltk ١ ׮ܲ ۄ੉࠳۞ܻ৬ э੉ ಞܻೞѱ ॶ ࣻ ੓ѱ ੋఠಕ੉झܳ ాੌ • ೠӖਸ ׮ܙ ٸ ೙ਃೠ пઙ utilਸ ୶оೣ 37
  17. 38.

    KoNLPy৬ ୐ߣ૩ PyCon KR • ׼द ౵੉ॆ झथ: "౵੉௑੉ ೠҴীࢲ

    ୊਺ ৌܽ؀!" • ա: "ա৬ ࠺तೠ ࠛಞೣਸ ѻҊ ੓ח ࢎۈ੉ ݆૑ ঋਸө?" • ٜ݅؍ Ѧ ੜ ಁః૚೧ࢲ ҕѐ೧ࠁ੗! 38
  18. 39.

    39

  19. 40.

    ࢎۈٜ਷ ৵ KoNLPyܳ ࢎਊ೮ਸө? • ୡࠁ੗: ࢎਊߨ੉ एਕࢲ? ؊ ए਋ݶ

    જѷ֎ਃ • ೟ࢤ: ࠁҊ ٮۄೡ ৘दо ੓যࢲ? • पޖ੗: ׮নೠ ҳഅ୓ р ࢿמ ࠺Үо ಞ೧ࢲ? • ৻Ҵੋ: ޙࢲо ৔য۽ب ॳৈ ੓যࢲ? • ജ҃੸ ਃੋ: ੉റ ౵੉ॆ੉ Ҵղ৻ীࢲ ೠହ ੋӝށ੉ܳ ೧ࢲ? 40
  20. 42.

    ੉ٞ೧ী فߣ૩ PyCon KR • ѐੋ੸ زӝ: ୐ ౵੉௑ ҃೷੉

    ցޖ જও׮ • ׼द ೟҅ীࢲח representation learning੉ ೥೧૓ द੼ • word2vec, doc2vecਸ Ҵղ੄ ౵੉ॆ ழޭפ౭ী ࣗѐ೧ࠅө? • ੘֙ী য়೑ࣗझ۽ KoNLPyܳ ҕѐ೮૑݅ ݄࢚ ష੉ؘ੉ఠо ߹۽ হ׮. ੌױ ೠҴয۽ ػ ؘ੉ఠࣇਸ ٜ݅যࠁ੗! 42
  21. 43.

    ೠҴয ৔ചಣ ؘ੉ఠ nsmc $ head ratings_train.txt id document label

    9976970 ই ؊ࡂ.. ૓૞ ૞ૐա֎ਃ ݾܻࣗ 0 3819312 ൙...ನझఠࠁҊ ୡ٬৔ച઴....য়ߡোӝઑର оߺ૑ ঋҳա 1 10265843 ցޖ੤㄂঻׮ӒېࢲࠁחѪਸ୶ୌೠ׮ 0 9045019 Үبࣗ ੉ঠӝҳݢ ..ࣛ૒൤ ੤޷ח হ׮..ಣ੼ ઑ੿ 0 6483659 ࢎ੉ހಕӒ੄ ੊࢓झ۠ োӝо ثࠁ৓؍ ৔ച!झ౵੉؊ݔীࢲ יযࠁ੉ӝ݅ ೮؍ ழझ౯ ؍झ౟о ցޖաب ੉࡜ࠁ৓׮ 1 5403919 ݄ Ѧ਺݃ ڏ 3ࣁࠗఠ ୡ١೟Ү 1೟֙ࢤੋ 8࢓ਊ৔ച.ƀƀƀ...߹߈ѐب ইө਑. 0 7797314 ਗ੘੄ ӟ੢хਸ ઁ؀۽ ࢓۰ղ૑ޅ೮׮. 0 9443947 ߹ ߈ѐب ইӰ׮ ਄աৡ׮ ੉਽҃ ӡਊ਋ োӝࢤഝ੉ݻ֙ੋ૑..੿݈ ߊ۽೧ب ӒѪࠁױ իѶ׮ ժ஖.хӘ݅߈ࠂ߈ࠂ.. 0 7156791 ঘ࣌੉ হחؘب ੤޷ ੓ח ݻউغח ৔ച 1 • ઱য૓ ৔ചಣਸ ӛ੿ ژח ࠗ੿ਵ۽ ࠙ܨೞח ؘ੉ఠࣇ • Maas et al. 2011੄ IMDB ؘ੉ఠࣇ(৔য)ਸ ߮஖݃௼ 43
  22. 49.

    ೠҴয য়೑ؘ੉ఠ അട: Sequence classification / labeling ҕѐदӝ ੉ܴ ݾ੸

    ௼ӝ ۄ੉ࣃझ 2007 ࣁઙ௏ಌझ ಿࢎకӦ 838k sentences CC BY-NC-ND 4.0 2012 KOSAC хࢿ࠙ܨ (хࢿয ࢎ੹ب ߓನ) 7.7k sentences Custom 2015 nsmc хࢿ ࠙ܨ 200k sentences (Train: 150k, Test: 50k) CC0 (Public Domain) 2016 KoreanNERCorpus NER 3.5k sentences - 2018 nlp-challenge NER 90k sentences - 2018 nlp-challenge SRL 35k sentences - 2018 Question pair Paraphrase detection 7k sentence pairs - ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ 49
  23. 50.

    ೠҴয য়೑ؘ੉ఠ അട: Sequence generation ҕѐदӝ ੉ܴ ݾ੸ ௼ӝ ۄ੉ࣃझ

    2015 JPO patent corpus ӝ҅ߣ৉ ja-ko, ౠೲ بݫੋ 257k sentence pairs Custom (࢚স੸ ੉ਊ ࠛо) 2017 Korean parallel corpora ӝ҅ߣ৉ ko-{en, fr, ja} п ঱যह߹ 0.7k, 95k, 0.2k sentence pairs CC BY-NC-ND 3.0 (࢚স੸ ࢎਊ ࠛо) 2018 KSS ਺ࢿ ೤ࢿ/ੋध 12.9k pairs CC BY-NC-SA 4.0 (࢚স੸ ੉ਊ ࠛо) 2018 Chatbot data ؀ച 12k pairs - 2018 OpenSubtitles2018 ӝ҅ߣ৉ ko-* 1.3m sentence pairs (For ko-en only) - 2019 AIHub ೠҴয-৔য ߣ৉ ߽۳ ݈ޡ஖ ӝ҅ߣ৉ ko-en 16k sentence pairs (To be 1.6m within 2019) - ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ "*)VC੄҃਋ۄ੉ࣃझоݺदغয੓૑ঋ૑݅ ઺ࣗӝস ߮୊ӝস झఋ౟স ѐੋѐߊ੗ োҳ੗١੉ఋѶਬ੷ੋ੿ࠗࢎস੉޲۽CC0ژחCC BY੉૑ঋਸө୶ஏ೧ࠇפ׮ ژ ઁоӝ҅ߣ৉ীҙब੉੓যࢲ׮ܲ࠙ঠী࠺೧࢚؀੸ਵ۽؊աৌೡࣻ੓חѪэ਷ؘ֬஘જ਷ؘ੉ఠࣇ੉੓׮ݶBXFTPNFLPSFBOOMQэ਷ܻझ౟ীҕਬ೧઱ࣁਃ 50
  24. 51.

    ೠҴয য়೑ؘ੉ఠ അട: Others ҕѐदӝ ੉ܴ ݾ੸ ௼ӝ ۄ੉ࣃझ 2018

    KorQUAD MRC 66k Q/A pairs - 2019? AIHub ӝ҅ة೧ MRC 450k Q/A pairs - 2019 cc-kedict ೠ৔ ࢎ੹ 13.7k entries CC BY-SA 3.0 2019 kosentences Self-supervised learning 31m sentences MIT + GNU Free Documentation + CC BY-NC-SA ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ ੉઺ݻо૑חઁо૒੽੉ਊ೧ࠁ૑ޅ೮णפ׮ 51
  25. 54.

    જ਷ ࢎ۹ 2: KSS https://kaggle.com/bryanpark/korean-single-speaker-speech-... • Ҵղ ୭ୡ੄ ਺ࢿ য়೑ؘ੉ఠ

    • ઁߨ ݆਷ ࠙۝ • ۄ੉ࣃझо ڢ۶ೞѱ ݺदغয ੓যࢲ ޖ঺ਸ ೡ ࣻ ੓Ҋ হח૑о ݺഛೣ 54
  26. 55.

    ৵ য়೑ؘ੉ఠо ઺ਃೠоਃ? 1. ߮஖݃௼о ؼ ࣻ ੓ӝ ٸޙ! •

    ML ݽ؛ٜ਷ ؘ੉ఠ৬ ૑಴о эইঠ ݽ؛ р ࢲ۽ ࠺Үо оמ • ࠺Үо оמ೧૑ݶ ӝࣿ ߊ੹੉ ৡ׮ 2. ־ҳա ߄۽ ࠙ࢳ੉ա ݽ؛݂ਸ द੘ೡ ࣻ ੓ӝ ٸޙ! • ೐۽Ӓې߁ীࢲ reinventing the wheel੉ ઙઙ ҃҅੄ ؀࢚੉ غ٠ • ݽفо ؘ੉ఠ ஂٙҗ ੿ઁܳ ೡ ೙ਃח হਗ਼ইਃ 55
  27. 56.

    ؘ੉ఠܳ ҕѐೡ ٸ ഛੋೞݶ જ਷ ੼ 1. ࢎਊ੗о ߄۽ ׮਍۽٘೧ࢲ

    ࢎਊೡ ࣻ ੓חо? • оә੸੉ݶ ഥਗоੑ, ੉ਊز੄ ੺ର হ੉ ߄۽ ࢎਊೡ ࣻ ੓ѱ ೣ 2. ਗޙী ѐੋ੿ࠁ/੷੘ӂ ޙઁח হחо? • য়೑ؘ੉ఠח ӝࣿ ߊ੹ਸ ਤ೧ ݒ਋ ઺ਃೞ૑݅ ѐੋ੿ࠁ৬ ੷੘ӂب ઓ઺߉Ҋ ૑ெઉঠ ೣ! 3. оә੸੉ݶ ۄ੉ࣃझܳ ԙ ݺद೧઱ࣁਃ! • ੉ਊ੗੄ ߹ب੄ ޙ੄হ੉ оמೠ Ѫ, ࠛоמೠ Ѫਸ ঌܻӝ ਤೣ 56
  28. 57.

    য়೑ؘ੉ఠܳ ࢎਊೡ ٸ ഛੋೞݶ જ਷ ੼ 1. ؘ੉ఠо ୽࠙൤ ௾о?

    ୽࠙൤ ੄޷੓ח Ѿҗܳ յ ࣻ ੓ח ੿ب۽ ୽࠙ೠо? 2. ۄ੉ࣃझо ޖ঺ੋо? ੤ߓನо оמೠо? ࢚সਊਵ۽ ੉ਊ೧ب غחо? 57
  29. 58.

    Call for partipation • ೠҴয NLP э੉ ߊ੹दெਃ! • য়೑ࣗझ۽ب,

    য়೑ؘ੉ఠ۽ب ӝৈೡ ࣻ ੓णפ׮ • য়೑ࣗझח ౵੉ॆ੉যب જ૑݅ ԙ ౵੉ॆ਷ ইפযب ؾפ׮ • য়೑ؘ੉ఠח ੿ࠗ, ӝস, ೟Ү ݽف ೣԋ೧ਃ ޛۿ ѐੋ ѐߊ੗ب • ೠҴয NLP ݈Ҋ ׮ܲ ੤߀ח ࠙ঠب ݆੉ ӝৈೞҊ ࢲ۽ ҕਬ೧ਃ! 58