Upgrade to Pro — share decks privately, control downloads, hide ads and more …

파이썬과 커뮤니티와 한국어 오픈데이터

Lucy Park
August 17, 2019

파이썬과 커뮤니티와 한국어 오픈데이터

PyCon Korea 2019 키노트 발표 자료입니다.

Lucy Park

August 17, 2019
Tweet

Other Decks in Programming

Transcript

  1. ߅਷੿ a.k.a. lucypark, echojuliett, e9t • ӝ҅ߣ৉ਸ ೞח ݠन۞׬ ূ૑פয

    • ؘ੉ఠ۽ ૑ध੄ ੢߷ਸ ծ୶Ҋ र਷ ࢎۈ • Yak shaver 2
  2. ٸח 2019֙ 5ਘ • ӝഥо ػ׮ݶ ೞҊ र਷ ݈੉ ੓঻য!

    • "ઁউ೧઱࣊ࢲ хࢎ೤פ׮! যڃ ফӝܳ ೧ঠೡ૑ ഛन੉ ٜ૑ח ঋ૑݅ ਬ ੊ೠ ফӝܳ ೡ ࣻ ੓ب۾ ળ࠺೧ࠅѱਃ." • ߊ಴ө૑ 3ѐਘ੉ա ੓ਵפ ળ࠺ೡ दрب ୽࠙ೞ׮! 3
  3. ԝԝೠ ҅ദ • ೠҴয NLP੄ য়೑ࣗझ/ؘ੉ఠী ؀ೠ ࢲ߬੉ܳ ೧ࢲ, അടਸ

    ҕਬೞҊ ׮ э੉ ߊ੹दெաо੗Ҋ ઁউ೧ঠѷয! • ః֢౟ۄݶ ੿ࠁܳ ੹׳ೞӝࠁ׮ח ನਊinclusion , ׮নࢿdiversity , ழޭפ౭৬ э਷ о஖ܳ ࠗпೞח ߊ಴ݶ જਸ Ѫ эই • Ӓؘ۠ ژ ౵੉௑੉פө ౵੉ॆী ؀ೠ ফӝب ೧ঠ૑ • Ӓۧ׮ݶ 3֙݅ী konlpyب ખ সؘ੉౟ ೧ࠅө? пઙ ؘ੉ఠࣇਸ konlpy.downloadী ನೣ೧ࢲ ࢎۈٜ੉ औѱ ׮਍۽٘ ߉ѱ ೞݶ જਸ Ѫ э׮! 4
  4. ӒܻҊ 2019֙ 8ਘ • konlpyח Ҋசѱ ցޖ ݆׮... э੉ ٜ݅যоਃ!

    • ই૒ ߊ಴੗ܐب হ֎... ƕƕ • ׮೯൤ ੉ܲ ইஜ੄ ୐ ߊ಴פө ࢎۈ੉ ߹۽ উৢѢঠ... 5
  5. য়ט ೡ ੉ঠӝ • ޙઁ ੄ध ҕਬ • ழޭפ౭ী ੘ѱա݃

    ӝৈೡ ࣻ ੓঻؍ ѐੋ੸ੋ ҃೷ ҕਬ ঢ়զ ফӝ • ೠҴয য়೑ؘ੉ఠ ੉ঠӝب ઑӘ ઱߸ ࢎۈٜ਷ ݔզ ٜ঻؍ ফӝ *NBHFTPVSDF4BOESBBOE8PP 3JDIBSETHVJEFUPTPGUXBSFEFWFMPQNFOU  6
  6. ೐۽Ӓې߁਷ ৵ ೞחо? • ஂ޷ • ݡҊࢎפ્ • ѐੋ੄ ࢿ੢

    • ա੄ ழޭפ౭ী ӝৈೞӝ ੉೦ݾٜ਷࢚ഐߓఋ੸੉૑ঋ׮ 8
  7. 11

  8. 12

  9. ౱ನಯ http://popong.com • ৘࠺ ٣੗੉ց, ѐߊ੗, ӝദ੗ ١ਵ۽ ҳࢿػ ೟ࢤ

    10ৈ ݺ • ӝࣿ۽ॄ ؀ೠ޹Ҵ ੿஖ܳ ٍൔ٘חѱ ݾ಴ • ૒੽ ࢲ࠺झܳ ٜ݅যࠁݶࢲ ਋ܻо ࢿ੢ೞח Ѫ਷ ؒ! (ژח ؊ ௾ ݾ੸) 1010/(1VCMJD0QFO10MJUJDBMFOHJOFFSJ/( 19
  10. ݒ઱ షਃੌ݃׮ ݽৈ ӝദҗ షۿҗ झఠ٣ ؀׮ࣻо ੑߨҗ੿ ؀೧ ইחѱ

    ೞաب হ঻Ѣٚਃ Ӓۧѱ 2ߣ੄ ௼ܻझ݃झܳ ೣԋ ݏ੉ೠ റ... 22
  11. ਢࢲ࠺झ "؀ೠ޹Ҵ ੿஖੄ ݽٚ Ѫ" ఍ࢤ ੉Ѫ ٸޙী ௼܀݂ب ৌब൤

    ೞҊ ইऔѱب૑Ә਷ബਊ؀࠺࠺ਊ੉݆੉૑୹ػ׮Ҋ౸ױ೧ࢲ֙ਘࠗఠPVUPGTFSWJDF ೞ૑݅౱ನಯӝо׮दबತࣗࢤ઺੉ۄ׮दࣁ࢚ীաৢࣻب੓যਃ 23
  12. Ҵഥ੄ਗ/੄উ ؘ੉ఠ REST API • ࢎप ਢࢲ࠺झח ࣳா੉झ • ੉ѱ

    ਗې ૓೯ೞ۰؍ ೐۽ં౟ • ؊ ݆਷ ࢎۈٜ੉ ੿஖ ؘ੉ఠܳ ࠙ࢳೞѢա ഝਊ೧ࢲ ࢲ࠺झܳ ٜ݅ӝܳ ߄ۗ਺ ইऔѱب૑Ә਷੉ѪبPVUPGTFSWJDFҊ௏٘݅թই੓૑݅ ੉ઁחҴഥীࢲ૒੽ؘ੉ఠܳઁҕ೤פ׮ ഥ੄۾ ੄উ ੄ਗ 28
  13. ؀ೠ޹Ҵ ੿஖ܳ ٍൔٜѷ׮Ҋ ࢤпೠ ಁӝ ֈ஖ח ୒ٜ֙਷ • ೠಞਵ۽ח ઱߸੄

    ੘਷ ࢎഥ۽ࠗఠ ੋ੿ਸ ߉ӝب ೮૑݅ • ࣁ࢚਷ Ӓۧѱ औѱ ߸ೞחѱ ইפۄח Ѫਸ ߓਛҊ ࢲߡ ࠺ਊ੉ ࠺ऱ׮חѪب ঌѱ غҊ • ޖ঺ࠁ׮ب ࣗ઺ೠ જ਷ ஘ҳܳ ঳Ҋ 30
  14. Ӓ যڃ ҃೷ࠁ׮ ч૓ જ਷ ѐߊ ޙചܳ ߓਛणפ׮ • ޙࢲച

    ೧઱ࣁਃ. ౱ ղ৻৬ ࣗాೞח ߑߨ • Git੄ જ਷ ೐ۑ౭झח ੉Ѣېਃ. ౱ ղ৻৬ ࣗాೡ ٸ ૑ఃݶ જ਷ ݒց • ࢜ ೐۽ં౟ীࢲח Flaskۄח ೐ۨ੐ਕ௼ܳ ॄࠅөਃ? ࢜۽਍ بҳܳ ੊൤ח ߑߨ • ই੉٣যח working prototypeਵ۽ ࠁৈ઱ࣁਃ. ఌ࢚ҕۿਸ ߩযա ౱ਸ ਑૒੉ח ߑߨ • Ӓ ௏٘/ؘ੉ఠ ۄ੉ࣃझо ޤীਃ? ఋੋ੄ ੷੘ޛਸ ઓ઺ೞח ੗ࣁ 31
  15. ٸח 2014֙ • ઁ ੹ҕ੉ ਗې ఫझ౟ی਷ ࢚ҙহҊ ं੗ܳ ࠙ࢳೞח

    ؘ੉ఠ݃੉׬ • ೟Ү ೐۽ં౟ܳ ࣻ೯ೞ؍ ઺ ೠҴয ࠙ࢳਸ ಞܻೞѱ ೞӝ ਤ೧ द੘ • ೠҴয ࠙ࢳਸ ೞ۰ݶ ੌױ ష௼ա੉૚ਸ ೧ঠೞחؘ যڌѱ ೞ૑? • ഋకࣗ ࠙ࢳਸ ೞݶ ػ׮חؘ য়೑ࣗझח ੓ա? • ݻ о૑о ੓ח Ѫ э਷ؘ ࢿמ਷ যڌѱ ׮ܰ૑? 36
  16. Enter KoNLPy >>> from konlpy.tag import Okt >>> okt =

    Okt() >>> okt.pos('݅աࢲ ߈оਕਃ!') [('݅աࢲ', 'Verb'), ('߈оਕਃ', 'Adjective'), ('!', 'Punctuation')] • пઙ য়೑ࣗझ ഋకࣗ ࠙ࢳӝܳ ݽ਺ • nltk ١ ׮ܲ ۄ੉࠳۞ܻ৬ э੉ ಞܻೞѱ ॶ ࣻ ੓ѱ ੋఠಕ੉झܳ ాੌ • ೠӖਸ ׮ܙ ٸ ೙ਃೠ пઙ utilਸ ୶оೣ 37
  17. KoNLPy৬ ୐ߣ૩ PyCon KR • ׼द ౵੉ॆ झथ: "౵੉௑੉ ೠҴীࢲ

    ୊਺ ৌܽ؀!" • ա: "ա৬ ࠺तೠ ࠛಞೣਸ ѻҊ ੓ח ࢎۈ੉ ݆૑ ঋਸө?" • ٜ݅؍ Ѧ ੜ ಁః૚೧ࢲ ҕѐ೧ࠁ੗! 38
  18. 39

  19. ࢎۈٜ਷ ৵ KoNLPyܳ ࢎਊ೮ਸө? • ୡࠁ੗: ࢎਊߨ੉ एਕࢲ? ؊ ए਋ݶ

    જѷ֎ਃ • ೟ࢤ: ࠁҊ ٮۄೡ ৘दо ੓যࢲ? • पޖ੗: ׮নೠ ҳഅ୓ р ࢿמ ࠺Үо ಞ೧ࢲ? • ৻Ҵੋ: ޙࢲо ৔য۽ب ॳৈ ੓যࢲ? • ജ҃੸ ਃੋ: ੉റ ౵੉ॆ੉ Ҵղ৻ীࢲ ೠହ ੋӝށ੉ܳ ೧ࢲ? 40
  20. ੉ٞ೧ী فߣ૩ PyCon KR • ѐੋ੸ زӝ: ୐ ౵੉௑ ҃೷੉

    ցޖ જও׮ • ׼द ೟҅ীࢲח representation learning੉ ೥೧૓ द੼ • word2vec, doc2vecਸ Ҵղ੄ ౵੉ॆ ழޭפ౭ী ࣗѐ೧ࠅө? • ੘֙ী য়೑ࣗझ۽ KoNLPyܳ ҕѐ೮૑݅ ݄࢚ ష੉ؘ੉ఠо ߹۽ হ׮. ੌױ ೠҴয۽ ػ ؘ੉ఠࣇਸ ٜ݅যࠁ੗! 42
  21. ೠҴয ৔ചಣ ؘ੉ఠ nsmc $ head ratings_train.txt id document label

    9976970 ই ؊ࡂ.. ૓૞ ૞ૐա֎ਃ ݾܻࣗ 0 3819312 ൙...ನझఠࠁҊ ୡ٬৔ച઴....য়ߡোӝઑର оߺ૑ ঋҳա 1 10265843 ցޖ੤㄂঻׮ӒېࢲࠁחѪਸ୶ୌೠ׮ 0 9045019 Үبࣗ ੉ঠӝҳݢ ..ࣛ૒൤ ੤޷ח হ׮..ಣ੼ ઑ੿ 0 6483659 ࢎ੉ހಕӒ੄ ੊࢓झ۠ োӝо ثࠁ৓؍ ৔ച!झ౵੉؊ݔীࢲ יযࠁ੉ӝ݅ ೮؍ ழझ౯ ؍झ౟о ցޖաب ੉࡜ࠁ৓׮ 1 5403919 ݄ Ѧ਺݃ ڏ 3ࣁࠗఠ ୡ١೟Ү 1೟֙ࢤੋ 8࢓ਊ৔ച.ƀƀƀ...߹߈ѐب ইө਑. 0 7797314 ਗ੘੄ ӟ੢хਸ ઁ؀۽ ࢓۰ղ૑ޅ೮׮. 0 9443947 ߹ ߈ѐب ইӰ׮ ਄աৡ׮ ੉਽҃ ӡਊ਋ োӝࢤഝ੉ݻ֙ੋ૑..੿݈ ߊ۽೧ب ӒѪࠁױ իѶ׮ ժ஖.хӘ݅߈ࠂ߈ࠂ.. 0 7156791 ঘ࣌੉ হחؘب ੤޷ ੓ח ݻউغח ৔ച 1 • ઱য૓ ৔ചಣਸ ӛ੿ ژח ࠗ੿ਵ۽ ࠙ܨೞח ؘ੉ఠࣇ • Maas et al. 2011੄ IMDB ؘ੉ఠࣇ(৔য)ਸ ߮஖݃௼ 43
  22. ೠҴয য়೑ؘ੉ఠ അട: Sequence classification / labeling ҕѐदӝ ੉ܴ ݾ੸

    ௼ӝ ۄ੉ࣃझ 2007 ࣁઙ௏ಌझ ಿࢎకӦ 838k sentences CC BY-NC-ND 4.0 2012 KOSAC хࢿ࠙ܨ (хࢿয ࢎ੹ب ߓನ) 7.7k sentences Custom 2015 nsmc хࢿ ࠙ܨ 200k sentences (Train: 150k, Test: 50k) CC0 (Public Domain) 2016 KoreanNERCorpus NER 3.5k sentences - 2018 nlp-challenge NER 90k sentences - 2018 nlp-challenge SRL 35k sentences - 2018 Question pair Paraphrase detection 7k sentence pairs - ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ 49
  23. ೠҴয য়೑ؘ੉ఠ അട: Sequence generation ҕѐदӝ ੉ܴ ݾ੸ ௼ӝ ۄ੉ࣃझ

    2015 JPO patent corpus ӝ҅ߣ৉ ja-ko, ౠೲ بݫੋ 257k sentence pairs Custom (࢚স੸ ੉ਊ ࠛо) 2017 Korean parallel corpora ӝ҅ߣ৉ ko-{en, fr, ja} п ঱যह߹ 0.7k, 95k, 0.2k sentence pairs CC BY-NC-ND 3.0 (࢚স੸ ࢎਊ ࠛо) 2018 KSS ਺ࢿ ೤ࢿ/ੋध 12.9k pairs CC BY-NC-SA 4.0 (࢚স੸ ੉ਊ ࠛо) 2018 Chatbot data ؀ച 12k pairs - 2018 OpenSubtitles2018 ӝ҅ߣ৉ ko-* 1.3m sentence pairs (For ko-en only) - 2019 AIHub ೠҴয-৔য ߣ৉ ߽۳ ݈ޡ஖ ӝ҅ߣ৉ ko-en 16k sentence pairs (To be 1.6m within 2019) - ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ "*)VC੄҃਋ۄ੉ࣃझоݺदغয੓૑ঋ૑݅ ઺ࣗӝস ߮୊ӝস झఋ౟স ѐੋѐߊ੗ োҳ੗١੉ఋѶਬ੷ੋ੿ࠗࢎস੉޲۽CC0ژחCC BY੉૑ঋਸө୶ஏ೧ࠇפ׮ ژ ઁоӝ҅ߣ৉ীҙब੉੓যࢲ׮ܲ࠙ঠী࠺೧࢚؀੸ਵ۽؊աৌೡࣻ੓חѪэ਷ؘ֬஘જ਷ؘ੉ఠࣇ੉੓׮ݶBXFTPNFLPSFBOOMQэ਷ܻझ౟ীҕਬ೧઱ࣁਃ 50
  24. ೠҴয য়೑ؘ੉ఠ അട: Others ҕѐदӝ ੉ܴ ݾ੸ ௼ӝ ۄ੉ࣃझ 2018

    KorQUAD MRC 66k Q/A pairs - 2019? AIHub ӝ҅ة೧ MRC 450k Q/A pairs - 2019 cc-kedict ೠ৔ ࢎ੹ 13.7k entries CC BY-SA 3.0 2019 kosentences Self-supervised learning 31m sentences MIT + GNU Free Documentation + CC BY-NC-SA ۄ੉ࣃझоউ੸ഃ੓ח઴਷ۄ੉ࣃझоݺदغয੓૑ঋѢաઁоޅ଺਷҃਋ੑפ׮ ੉઺ݻо૑חઁо૒੽੉ਊ೧ࠁ૑ޅ೮णפ׮ 51
  25. જ਷ ࢎ۹ 2: KSS https://kaggle.com/bryanpark/korean-single-speaker-speech-... • Ҵղ ୭ୡ੄ ਺ࢿ য়೑ؘ੉ఠ

    • ઁߨ ݆਷ ࠙۝ • ۄ੉ࣃझо ڢ۶ೞѱ ݺदغয ੓যࢲ ޖ঺ਸ ೡ ࣻ ੓Ҋ হח૑о ݺഛೣ 54
  26. ৵ য়೑ؘ੉ఠо ઺ਃೠоਃ? 1. ߮஖݃௼о ؼ ࣻ ੓ӝ ٸޙ! •

    ML ݽ؛ٜ਷ ؘ੉ఠ৬ ૑಴о эইঠ ݽ؛ р ࢲ۽ ࠺Үо оמ • ࠺Үо оמ೧૑ݶ ӝࣿ ߊ੹੉ ৡ׮ 2. ־ҳա ߄۽ ࠙ࢳ੉ա ݽ؛݂ਸ द੘ೡ ࣻ ੓ӝ ٸޙ! • ೐۽Ӓې߁ীࢲ reinventing the wheel੉ ઙઙ ҃҅੄ ؀࢚੉ غ٠ • ݽفо ؘ੉ఠ ஂٙҗ ੿ઁܳ ೡ ೙ਃח হਗ਼ইਃ 55
  27. ؘ੉ఠܳ ҕѐೡ ٸ ഛੋೞݶ જ਷ ੼ 1. ࢎਊ੗о ߄۽ ׮਍۽٘೧ࢲ

    ࢎਊೡ ࣻ ੓חо? • оә੸੉ݶ ഥਗоੑ, ੉ਊز੄ ੺ର হ੉ ߄۽ ࢎਊೡ ࣻ ੓ѱ ೣ 2. ਗޙী ѐੋ੿ࠁ/੷੘ӂ ޙઁח হחо? • য়೑ؘ੉ఠח ӝࣿ ߊ੹ਸ ਤ೧ ݒ਋ ઺ਃೞ૑݅ ѐੋ੿ࠁ৬ ੷੘ӂب ઓ઺߉Ҋ ૑ெઉঠ ೣ! 3. оә੸੉ݶ ۄ੉ࣃझܳ ԙ ݺद೧઱ࣁਃ! • ੉ਊ੗੄ ߹ب੄ ޙ੄হ੉ оמೠ Ѫ, ࠛоמೠ Ѫਸ ঌܻӝ ਤೣ 56
  28. য়೑ؘ੉ఠܳ ࢎਊೡ ٸ ഛੋೞݶ જ਷ ੼ 1. ؘ੉ఠо ୽࠙൤ ௾о?

    ୽࠙൤ ੄޷੓ח Ѿҗܳ յ ࣻ ੓ח ੿ب۽ ୽࠙ೠо? 2. ۄ੉ࣃझо ޖ঺ੋо? ੤ߓನо оמೠо? ࢚সਊਵ۽ ੉ਊ೧ب غחо? 57
  29. Call for partipation • ೠҴয NLP э੉ ߊ੹दெਃ! • য়೑ࣗझ۽ب,

    য়೑ؘ੉ఠ۽ب ӝৈೡ ࣻ ੓णפ׮ • য়೑ࣗझח ౵੉ॆ੉যب જ૑݅ ԙ ౵੉ॆ਷ ইפযب ؾפ׮ • য়೑ؘ੉ఠח ੿ࠗ, ӝস, ೟Ү ݽف ೣԋ೧ਃ ޛۿ ѐੋ ѐߊ੗ب • ೠҴয NLP ݈Ҋ ׮ܲ ੤߀ח ࠙ঠب ݆੉ ӝৈೞҊ ࢲ۽ ҕਬ೧ਃ! 58