pandasでのOSS活動事例と最初の一歩

@ PyConJP 2017 Keynote

Transcript

1. pandasͰͷOSS׆ಈ ࣄྫͱ࠷ॳͷҰา Masaaki Horikoshi @ ARISE analytics
2. • ງӽ ਅө @ גࣜձࣾ ARISE analytics • OSS׆ಈ • https://github.com/sinhrks ࣗݾ঺հ 867 2017/9/3ݱࡏ 11 106 15 9 2 2 3 77 286 1,879 10,699 ggfortify(R) pandas-ml Github Stars Org. Members (Committers) Contributors
3. ձࣾ঺հ • גࣜձࣾ ARISE analytics • 2017೥4݄ઃཱ • ۀ຿ • ௨৴αʔϏεɺEίϚʔεͳͲʹ͓͚Δσʔλ෼ੳ • ϨίϝϯυΤϯδϯ΍৽نιϦϡʔγϣϯͷ։ൃ
4. ໨࣍ • PythonͱՊֶܭࢉ • pandasͰͷOSS׆ಈࣄྫ • OSS׆ಈ ࠷ॳͷҰา
5. PythonͱՊֶܭࢉ
6. Pythonͱ͸ • ൚༻ϓϩάϥϛϯάݴޠ • ۙ೥ɺՊֶܭࢉͰ΋޿͘࢖ΘΕ͍ͯΔ • Google Trends (Python / ೔ຊ)
7. IEEE Spectrum The 2017 Top Programming Language http://spectrum.ieee.org/computing/software/the-2017-top-prog
8. Pythonͱ͸ • Guido van Rossum: • … The first sound bite I had for Python was, "Bridge the gap between the shell and C.” • So I never intended Python to be the primary language for programmers … It was intended to be a second language for people who were already experienced programmers… • We now have a large community of people using Python as an educational language … These people aren't and may never be professional programmers, but they still find some programming skills useful. • Python's Design Goals (A Conversation with Guido van Rossum, Part II) by Bill Venners 2003/01/20
9. ՊֶܭࢉͱPython • Python͕ՊֶܭࢉͰ࢖ΘΕ͍ͯΔཧ༝(ࢲݟ) • ڭҭػؔͰ࢖ΘΕ͍ͯΔ • C/Fortranࢿ࢈ͷར༻͕༰қ • ՊֶܭࢉܥOSS͕ൃలɾ੒ख़
10. Պֶܭࢉύοέʔδ
11. Պֶܭࢉύοέʔδ
12. ՊֶܭࢉܥίϛϡχςΟ • ࠃ಺֎Ͱ༷ʑͳίϛϡχςΟΠϕϯτ͕੝ΜʹߦΘ Ε͍ͯΔ • PyCon • PyData • SciPy • …
13. PyCon JP 2016 ࢀՃऀΞϯέʔτ • Q3. ීஈɺͲͷ෼໺ͰPythonΛ࢖͍ͬͯ·͔͢ʁ 168 127 111 94 55 51 14 12 12 2 0 20 40 60 80 100 120 140 160 180 &! / Private tools development Web+% / Web development #" */machine learning '( / Research & Development )$ / System management Big Data
       +% / Desktop Application   / Computer graphics +% / Game programming https://pycon.jp/2016/ja/files/32/pyconjp2016survey.htm
14. ւ֎ͷՊֶܭࢉܥΠϕϯτ • 2017೥7݄ SciPy 2017ʹࢀՃ • PythonΛ༻͍ͨσʔλ෼ੳɺՊֶݚڀʹؔ͢Δੈք࠷େ ن໛ͷΠϕϯτ • ࢀՃऀ: ໿700໊ • ظؒ: 9೔ؒ (νϡʔτϦΞϧɺεϓϦϯτؚ) • ։࠵৔ॴ: ςΩαεେֶ (ΦʔεςΟϯ)
15. SciPy2017ͷ༷ࢠ • Keynote: Drilling the Chicxulub Impact Structure / Sean Gulick, University of Texas • ڪཽઈ໓͸୯ҰͷᯁੴͷিಥͰͰ͖ͨՄೳੑ͕ߴ͍ͱ݁࿦ • σʔλղੳʹ͸PythonΛར༻
16. pandasͰͷOSS׆ಈࣄྫ
17. ͦ΋ͦ΋ pandas ͬͯԿʁ
18. pandasͱ͸ • σʔλ෼ੳͷͨΊͷߴ଎Ͱ࢖͍΍͍͢σʔλߏ଄Λఏڙ • 2࣍ݩදܗࣜ • RͰ͍͏“data.frame” • Author: Wes McKinney • License: BSD • ޠݯ: PANel DAta System
19. pandasͱ͸ import pandas as pd df = pd.read_csv(‘adult.csv’) df "EVMU
    %BUBTFU
    UBLFO
    GSPN
    6$*
    .-
    3FQPTJUPSZ
    -JDINBO
    .
     
    6$*
    .BDIJOF
    -FBSOJOH
    3FQPTJUPSZ
    <IUUQBSDIJWFJDTVDJFEVNM>
    *SWJOF
    $"
    6OJWFSTJUZ
    PG
    $BMJGPSOJB
    4DIPPM
    PG
    *OGPSNBUJPO
    BOE
    $PNQVUFS
    4DJFODF
    3FBE
    DTW
    pMF $PMVNOT *OEFY .JYFE
    EBUB
    UZQFT
20. pandasͱ͸ df[['age', 'marital-status']] df.groupby('income')['hours-per-week'].mean() (SPVQ
    CZ 4FMFDU "HHSFHBUF 4FMFDU
21. ͳͥ pandas ͕ඞཁͳͷʁ SQLͰΑ͘ͳ͍ʁ
22. ͳͥ pandas ͔ • ݱ࣮ͷ(Ԛ͍)σʔλΛɺ௚ײతʹૢ࡞Ͱ͖Δ
23. ͳͥ pandas ͔ • ྫ: খചళͰΩϟϯϖʔϯɾఱީ͕ചΓ্͛ʹ༩͑ΔӨڹΛ஌Γ͍ͨ • σʔλιʔε͕όϥόϥ… ೔࣌ ৔ॴ ఱީ 
     ౦ژ౎໨ࠇ۠ ੖Ε Ωϟϯϖʔϯ໊শ ։࢝೔ ऴྃ೔ 999   :::   Ωϟϯϖʔϯ͸ళฮ͝ͱʹ &YDFMͰه࿥ʜ ఱީσʔλ͸8FC͔Β$47Ͱ μ΢ϯϩʔυ ళฮ*% Ϣʔβ*% Ϩγʔτ*% ߪങ೔࣌ "" BBB  
     ## CCC  
     104σʔλΛ42-Ͱऔಘʜ
24. ͳͥ pandas ͔ • ྫ: খചళͰΩϟϯϖʔϯɾఱީ͕ചΓ্͛ʹ༩͑ΔӨڹΛ஌Γ͍ͨ • ༷ʑͳߟྀࣄ߲… ళฮ*% Ϣʔβ*% Ϩγʔτ*% ߪങ೔࣌ "" BBB  
     ## CCC  
     ೔࣌ ৔ॴ ఱީ 
     ౦ژ౎໨ࠇ۠ ੖Ε Ωϟϯϖʔϯ໊শ ։࢝೔ ऴྃ೔ 999   :::   ෼୯ҐͰूܭ͍ͨ͠ ৽ฉ޿ࠂͱళ಄νϥγͰ͸ޮ Ռ͕ҧ͏ʁ Ұఆ࣌ؒ͝ͱͷ؍ଌ஋Λ
    ద౰ͳϧʔϧͰඥ෇͚
    
25. pandas ͷҐஔ෇͚ CRISP-DM: Cross Industry Standard Process for Data Mining Ϗδωεͷཧղ σʔλͷཧղ σʔλ४උ ϞσϦϯά ධՁ ల։ ݱ࣮ͷσʔλΛཧղ͠ɺ
    ෼ੳͰ͖Δܗʹམͱ͢
26. pandas ΤίγεςϜ • I/O • pandas-datareader • pandas-msgpack • pandas-gbq • υϝΠϯ • geopandas • xarray • ػցֶश • sklearn-pandas • pandas-ml • ฒྻॲཧ • Dask • ։ൃऀ޲͚ • pandas-compat Լઢ͸pandas/pydataϦϙδτϦʹ͋Δ΋ͷ
27. ίϛολͬͯ ͲΜͳ͜ͱΛͯ͠Δʁ
28. pandas ͰͷOSS׆ಈܦҢ 1ϓϧϦΫΤετ=1ίϛοτӡ༻ Number of Commits ॳΊͯͷissue ·ͱ΋ʹ(?)׆ಈ։࢝ ίΞνʔϜՃೖ
29. ίϛολʹͳΔʹ͸ • ϓϩδΣΫτʹΑͬͯҧ͏͕ɺpandasͷ৔߹: • ࣭ɾྔͱ΋े෼ͳߩݙΛ͍ͯ͠Δ͜ͱ • Ұ೥Ҏ্׆ಈ͍ͯ͠Δ͜ͱ →ݱࡏͷίΞνʔϜϝϯόͷਪનɺ౤ථͰܾఆ
30. ίϛολͷ໾ׂ • Make decisions about: • The overall scope, vision and direction of the project. • Strategic collaborations with other organizations or individuals. • Specific technical issues, features, bugs and pull requests. • The services that are run by the project. • Regular community discussion doesn’t produce consensus.
31. ίϛολͷۀ຿ • Issue΁ͷճ౴ • ίʔυϨϏϡʔ • ϓϧϦΫΤετͷϚʔδ • पลΠϯϑϥ(CI౳)ͷ੔උɺӡ༻ • ϦϦʔε (ϦϦʔεϚωʔδϟ͕࣮ࢪ) • ςϨίϯ
32. ॏཁͳ͜ͱ • ϓϩμΫτͷ඼࣭ • ίϛϡχςΟͷ඼࣭ • ͜ΕΒΛҡ࣋͢ΔͨΊͷ࢓૊Έͮ͘Γ
33. ϓϩμΫτͷ඼࣭ • ࢖͍΍͍͢ • Πϯετʔϧ • ϢʔβϏϦςΟ • υΩϡϝϯτ • ඼࣭͕୲อ͞Ε͍ͯΔ • ܧଓతΠϯςάϨʔγϣϯ • ύϑΥʔϚϯεςετ
34. Πϯετʔϧ • Anaconda • ֤छ wheel ͷఏڙ
35. ϢʔβϏϦςΟ • API • ύοέʔδ಺֎Ͱͷ౷Ұੑ • ໊લۭؒΛద੾ʹ෼ׂ(ΞΫηα) • ޙํޓ׵ੑ • ௚ͪʹมߋͤͣඇਪ঑Խˠ2ϝδϟʔόʔδϣϯޙʹมߋ • աڈʹγϦΞϥΠζ͞ΕͨΦϒδΣΫτͷಡΈࠐΈΛ୲อ
36. υΩϡϝϯτ • ެࣜυΩϡϝϯτ • http://pandas.pydata.org/pandas-docs/stable/ • ॻ੶ • PythonʹΑΔσʔλ෼ੳೖ໳ / Wes McKinney • ༗ࢤυΩϡϝϯτ • Modern pandas / Tom Augspurger • https://tomaugspurger.github.io/modern-1.html
37. ܧଓతΠϯςάϨʔγϣϯ • ੩తνΣοΫ (flake8) • ࣗಈςετ (Travis-CI, AppVeyor, Circle-CI) • Python όʔδϣϯ x पล؀ڥͷ૊Έ߹Θͤ • ΧόϨοδνΣοΫ (Codecov)
38. ύϑΥʔϚϯεςετ • Airspeed Velocity • ࢦఆίϛοτͰςετˍίϛοτؒൺֱ • 1000↑ͷϕϯνϚʔΫΛ࣮ߦ http://pandas.pydata.org/speed/pandas
39. ॏཁͳ͜ͱ • ϓϩμΫτͷ඼࣭ • ίϛϡχςΟͷ඼࣭ • ͜ΕΒΛҡ࣋͢ΔͨΊͷ࢓૊Έͮ͘Γ
40. ίϛϡχςΟͷ඼࣭ • ΦʔϓϯͰ͋Δ͜ͱ • ࢀՃ͠΍͍͢ • αϙʔςΟϒͰ͋Δ
41. ࢀՃ͠΍͘͢͢ΔͨΊʹ • Φʔϓϯͳίϛϡχέʔγϣϯ (GitHub, Gitter) • ߦಈنൣ (Code Of Conduct) • ։ൃऀ޲͚υΩϡϝϯτ(GitHub Wiki)
42. GitHub • దٓλά෇͚͠ɺٞ࿦ʹࢀՃ͠΍͘͢ • IssueςϯϓϨʔτΛར༻͠ɺॳ৺ऀͰ΋Ϩ ϙʔτ͠΍͘͢
43. ߦಈنൣ (Code of Conduct) • ΨόφϯεͷͨΊͷGitHubϦϙδτϦΛ੔උ • https://github.com/pandas-dev/pandas- governance
44. ։ൃऀ޲͚υΩϡϝϯτ • ։ൃऀ޲͚ • ίϯτϦϏϡʔγϣϯΨΠυ • Gitͷ࢖͍ํ • ςετํ๏ • ίʔσΟϯάελΠϧ • ϦϦʔεϊʔτͷॻ͖ํ • Specialities • … • ϝϯςφ޲͚ • ϦϦʔε࣌ͷνΣοΫϦετ • υΩϡϝϯτެ։ํ๏ • …
45. ։ൃऀ޲͚υΩϡϝϯτ • Specialities • ػೳผͷ༗ࣝऀҰཡ(ίΞνʔϜҎ֎΋ؚ) • ؔ࿈͢Δ Issue / ϓϧϦΫΤετ͔Β௨஌Λ ඈ͹͢ • https://github.com/pandas-dev/pandas/ wiki/Specialities
46. ஂମɾاۀ͔Βͷࢧԉ • • Institutional Partners • Continuum Analytics • Two Sigma
47. OSS׆ಈ ࠷ॳͷҰา
48. OSS׆ಈͷϝϦοτ • ঝೝཉٻ͕ຬͨͤΔ • ༗ࣝऀͷίʔυϨϏϡʔʹΑΓεΩϧΞοϓͰ ͖Δ • ಺෦࣮૷͕ཧղͰ͖ɺޮ཰తͳίʔυ͕ॻ͚Δ • मਖ਼ΛϚελʹऔΓࠐΜͰ΋Β͏͜ͱͰɺݸਓ ͷϝϯςෛՙΛܰݮͰ͖Δ
49. OSS׆ಈΛ࢝ΊΔʹ͸ • ίʔυमਖ਼͚͕ͩOSS׆ಈͰ͸ͳ͍ • એ఻͢Δ (ϒϩάΛॻ͘ɺൃද͢Δ) • σʔλܥͷ৔߹ɺOSSΛ࢖ͬͨ෼ੳϊ΢ϋ΢Ͱ΋ • ࣭໰ (StackOverflow, GitHub Issue)ʹ౴͑Δ • IssueΛॻ͘ • ϓϧϦΫΤετΛૹΔ
50. OSS׆ಈΛ࢝ΊΔʹ͸ • Կʹߩݙ͢Δ͔ΛܾΊΔ • ݸਓϓϩδΣΫτͱͯ͠ • ࣗࣾϓϩμΫτΛOSSͱͯ͠ • ۀ຿Ͱ࢖͏OSSʹରͯ͠ • ༗໊ͳOSSʹରͯ͠
51. ϓϧϦΫΤετ͕ ૹΕΔΑ͏ʹͳΓ͍ͨʂ
52. ॳΊͯͷϓϧϦΫΤετ • ·ͣ͸ҰͭͷϓϩδΣΫτ/ػೳ͔Β • ϓϩηε/πʔϧͷशख़ʹ͸ίετ͕͔͔Δ • ίʔυͷؔ܎ΛಡΈղ͘ͷ͸೉͍͠
53. ॳΊͯͷϓϧϦΫΤετ • ؆୯ͳ΋ͷ͔Β • υΩϡϝϯτͷվగ • Τϥʔϝοηʔδͷվળ • қ͍͠όάमਖ਼
54. ϓϧϦΫΤετͷྲྀΕ • ௚͍ͨ͠ issue Λ୳͢ • issueʹͳ͍৔߹ɺ·ͣissueΛॻ͍ͨํ͕ྑ͍ • ίʔυΛमਖ਼͢Δ • ϓϧϦΫΤετΛૹΔ
55. ϓϧϦΫΤετର৅ͷ୳͠ํ • େ͖ΊͷϓϩμΫτͷ৔߹ɺλάͰ੔ཧ͞Ε͍ͯ Δ • ෼໺΍೉қ౓Ͱର৅Λ୳͢ • ଞͷϢʔβ͕ணख͍ͯ͠Δ৔߹͸ආ͚ͨํ͕ྑ͍
56. ϓϧϦΫΤετͷૹΓํ • ࠷௿ݶͷϧʔϧΛकΔ • ໌ࣔ͞Ε͍ͯΔ΋ͷ͚ͩͰྑ͍ • ςετΛ௥Ճ͢Δ • ίʔσΟϯάن໿ΛकΔ • …
57. ϓϧϦΫΤετΛૹͬͨޙ • ։ൃऀ΍ଞϢʔβʹΑΔϨϏϡʔ • (ϓϩδΣΫτʹΑΓ)υΩϡϝϯτɺϦϦʔεϊʔτͷ੔උ • Ϛʔδ
58. ϋʔυϧʹײ͡ΔཁҼ • ӳޠ • ίϛϡχςΟϩʔΧϧϧʔϧ • Gitͷ࢖͍ํ • ٕज़ྗ • Կ͔໰୊͕͋Ε͹ڭ͑ͯ͘ΕΔͷͰɺա౓ʹؾΛ࢖͏ඞཁ͸ͳ͍
59. ϓϧϦΫΤετʹ͸ ͲΜͳ΋ͷ͕͋Δʁ
60. ϓϧϦΫΤετͷछྨ • υΩϡϝϯτमਖ਼ • ෆ۩߹मਖ਼ • ػೳ௥ՃɺAPIมߋ • ϦϑΝΫλϦϯά • ςετ௥Ճ • …
61. ϓϧϦΫΤετࣄྫ (1/3) υΩϡϝϯτमਖ਼ • मਖ਼಺༰Λ؆ܿʹهࡌ https://github.com/pandas-dev/pandas/pull/13312 ͜ͷυΩϡϝϯτɺͳΜ͔յΕͯΔͬΆ͍
62. ϓϧϦΫΤετࣄྫ (1/3) υΩϡϝϯτमਖ਼ ۭനߦΛೖΕ͚ͨͩ
63. • ෳࡶͳมߋͷ৔߹͸मਖ਼લޙͷڍಈΛ੔ཧ͢ Δ ࠓͷڍಈ͸
    ʜ
    ͕ͩ
    मਖ਼ޙͷڍಈ͸
    ʜ
    ʹͳΔɻ
    ͜Ε͸ʜ
    ͱ͍͏ϧʔϧʹج͍͍ͮͯͯʜ
    https://github.com/pandas-dev/pandas/pull/13849 ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ
64. • ϨϏϡʔࢦఠΛड͚ͭͭɺ࢓༷ΛݻΊΔ 999ͷ࣌ʹΤϥʔग़ͨ͠ํ͕Α͘ͳ͍ʁ ͦ͏ͳΜ͚ͩͲɺ999Λېࢭ͢Δͱผͷػೳ͕ಈ͔ͳ͘ͳΔ ͡Ό͋࢓ํͳ͍ͶɻଞͷΈΜͳ͸Ͳ͏ࢥ͏ʁ Ҏ߱ٞ࿦ ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ
65. • ϦϦʔεϊʔτ΍υΩϡϝϯτΛमਖ਼͢Δ ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ
66. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • ҙਤ͕఻ΘΒͳ͚Ε͹ίʔυΛࣔ͢ https://github.com/pandas-dev/pandas/pull/13312 ͍΍ɺผͷͱ͜௚ͨ͠ํ͕͍͍Μ͡Όͳ͍ʁ ӳޠ͕͓͔͍͔͠΋ʜɻ
    मਖ਼ՕॴΛϓϧϦΫͰૹͬͨͷͰ֬ೝͯ͠ ࠓͷڍಈ͸
    ʜ
    Ͱ
    मਖ਼ޙͷڍಈ͸
    ʜ
    ʹͳΔɻ
    ͨͿΜ
    ʜ
    ͷॲཧΛमਖ਼͢Ε͹͍͍
67. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • (Өڹൣғ͕େ͖͍৔߹)ͻͨ͢ΒςετΛ଍͢ ҟͳΔΫϥε͕ಉ͡ڍಈʹͳ Δ͔νΣοΫ
68. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • (Өڹൣғ͕େ͖͍৔߹)ͻͨ͢ΒςετΛ଍͢ ܕͷ૊Έ߹ΘͤΛ໢ཏతʹ νΣοΫ ͦΕͧΕͷܕͷ૊Έ߹Θ͕ͤ /VN1Zͱಉ͡ڍಈʹͳΔ͔ νΣοΫ
69. ؾΛ͚͍ͭͯΔ͜ͱ • ίϛϡχέʔγϣϯ • ӳޠ͸௨͡ͳ͍͔΋͠Εͳ͍͕ɺίʔυͳΒ఻ΘΔ • ίʔυमਖ਼ • ہॴతͳमਖ਼Λආ͚Δ • ςετΛͪΌΜͱॻ͘ • ࣗ෼Ҏ֎͕ͦͷίʔυΛ৮ͬͨࡍ΋όάͷຒΊࠐΈΛ༧๷Ͱ͖Δ͔ʁ • υΩϡϝϯτ΋Ͱ͖Δ͚ͩॻ͘ • (࢓༷͕͔ͬ͠Γ఻ΘΕ͹)୭͔௚ͯ͘͠ΕΔ
70. ΍ͬͺΓͪΐͬͱ େมͦ͏…ʁ
71. ࠓ೔͔ΒͰ͖Δ͜ͱ • ؾʹͳΔϓϩδΣΫτΛWatchͯ͠ΈΔ • ίϛϡχςΟͷงғؾ͕Θ͔Δ • ໘നͦ͏ͳ Issue / ϓϧϦΫΤετ͕͋Δ͔΋ʁ
72. ·ͱΊ
73. ·ͱΊ • PythonͱՊֶܭࢉ • pandasͰͷOSS׆ಈࣄྫ • OSS׆ಈ ࠷ॳͷҰา • PythonՊֶܭࢉίϛϡχςΟʹࢀՃͯ͠ΈΑ͏ • ؾʹͳΔϓϩδΣΫτΛWatchͯ͠ΈΑ͏