pandasでのOSS活動事例と最初の一歩

Transcript

1. pandasͰͷOSS׆ಈ ࣄྫͱ࠷ॳͷҰา Masaaki Horikoshi @ ARISE analytics

2. • ງӽ ਅө @ גࣜձࣾ ARISE analytics • OSS׆ಈ •

   https://github.com/sinhrks ࣗݾ঺հ 867 2017/9/3ݱࡏ 11 106 15 9 2 2 3 77 286 1,879 10,699 ggfortify(R) pandas-ml Github Stars Org. Members (Committers) Contributors

3. ձࣾ঺հ • גࣜձࣾ ARISE analytics • 2017೥4݄ઃཱ • ۀ຿ •

   ௨৴αʔϏεɺEίϚʔεͳͲʹ͓͚Δσʔλ෼ੳ • ϨίϝϯυΤϯδϯ΍৽نιϦϡʔγϣϯͷ։ൃ

4. ໨࣍ • PythonͱՊֶܭࢉ • pandasͰͷOSS׆ಈࣄྫ • OSS׆ಈ ࠷ॳͷҰา

5. PythonͱՊֶܭࢉ

6. Pythonͱ͸ • ൚༻ϓϩάϥϛϯάݴޠ • ۙ೥ɺՊֶܭࢉͰ΋޿͘࢖ΘΕ͍ͯΔ • Google Trends (Python /

   ೔ຊ)

7. IEEE Spectrum The 2017 Top Programming Language http://spectrum.ieee.org/computing/software/the-2017-top-prog

8. Pythonͱ͸ • Guido van Rossum: • … The first sound

   bite I had for Python was, "Bridge the gap between the shell and C.” • So I never intended Python to be the primary language for programmers … It was intended to be a second language for people who were already experienced programmers… • We now have a large community of people using Python as an educational language … These people aren't and may never be professional programmers, but they still find some programming skills useful. • Python's Design Goals (A Conversation with Guido van Rossum, Part II) by Bill Venners 2003/01/20

9. ՊֶܭࢉͱPython • Python͕ՊֶܭࢉͰ࢖ΘΕ͍ͯΔཧ༝(ࢲݟ) • ڭҭػؔͰ࢖ΘΕ͍ͯΔ • C/Fortranࢿ࢈ͷར༻͕༰қ • ՊֶܭࢉܥOSS͕ൃలɾ੒ख़

10. Պֶܭࢉύοέʔδ

11. Պֶܭࢉύοέʔδ

12. ՊֶܭࢉܥίϛϡχςΟ • ࠃ಺֎Ͱ༷ʑͳίϛϡχςΟΠϕϯτ͕੝ΜʹߦΘ Ε͍ͯΔ • PyCon • PyData • SciPy

    • …

13. PyCon JP 2016 ࢀՃऀΞϯέʔτ • Q3. ීஈɺͲͷ෼໺ͰPythonΛ࢖͍ͬͯ·͔͢ʁ 168 127 111

    94 55 51 14 12 12 2 0 20 40 60 80 100 120 140 160 180 &! / Private tools development Web+% / Web development #" */machine learning '( / Research & Development )$ / System management Big Data
       +% / Desktop Application   / Computer graphics +% / Game programming https://pycon.jp/2016/ja/files/32/pyconjp2016survey.htm

14. ւ֎ͷՊֶܭࢉܥΠϕϯτ • 2017೥7݄ SciPy 2017ʹࢀՃ • PythonΛ༻͍ͨσʔλ෼ੳɺՊֶݚڀʹؔ͢Δੈք࠷େ ن໛ͷΠϕϯτ • ࢀՃऀ:

    ໿700໊ • ظؒ: 9೔ؒ (νϡʔτϦΞϧɺεϓϦϯτؚ) • ։࠵৔ॴ: ςΩαεେֶ (ΦʔεςΟϯ)

15. SciPy2017ͷ༷ࢠ • Keynote: Drilling the Chicxulub Impact Structure / Sean

    Gulick, University of Texas • ڪཽઈ໓͸୯ҰͷᯁੴͷিಥͰͰ͖ͨՄೳੑ͕ߴ͍ͱ݁࿦ • σʔλղੳʹ͸PythonΛར༻

16. pandasͰͷOSS׆ಈࣄྫ

17. ͦ΋ͦ΋ pandas ͬͯԿʁ

18. pandasͱ͸ • σʔλ෼ੳͷͨΊͷߴ଎Ͱ࢖͍΍͍͢σʔλߏ଄Λఏڙ • 2࣍ݩදܗࣜ • RͰ͍͏“data.frame” • Author: Wes

    McKinney • License: BSD • ޠݯ: PANel DAta System

19. pandasͱ͸ import pandas as pd df = pd.read_csv(‘adult.csv’) df "EVMU
    %BUBTFU
    UBLFO
    GSPN
    6$*
    .-
    3FQPTJUPSZ
    

    -JDINBO
    .
     
    6$*
    .BDIJOF
    -FBSOJOH
    3FQPTJUPSZ
    <IUUQBSDIJWFJDTVDJFEVNM>
    *SWJOF
    $"
    6OJWFSTJUZ
    PG
    $BMJGPSOJB
    4DIPPM
    PG
    *OGPSNBUJPO
    BOE
    $PNQVUFS
    4DJFODF
    3FBE
    DTW
    pMF $PMVNOT *OEFY .JYFE
    EBUB
    UZQFT

20. pandasͱ͸ df[['age', 'marital-status']] df.groupby('income')['hours-per-week'].mean() (SPVQ
    CZ 4FMFDU "HHSFHBUF 4FMFDU

21. ͳͥ pandas ͕ඞཁͳͷʁ SQLͰΑ͘ͳ͍ʁ

22. ͳͥ pandas ͔ • ݱ࣮ͷ(Ԛ͍)σʔλΛɺ௚ײతʹૢ࡞Ͱ͖Δ

23. ͳͥ pandas ͔ • ྫ: খചళͰΩϟϯϖʔϯɾఱީ͕ചΓ্͛ʹ༩͑ΔӨڹΛ஌Γ͍ͨ • σʔλιʔε͕όϥόϥ… ೔࣌ ৔ॴ

    ఱީ 
     ౦ژ౎໨ࠇ۠ ੖Ε Ωϟϯϖʔϯ໊শ ։࢝೔ ऴྃ೔ 999   :::   Ωϟϯϖʔϯ͸ళฮ͝ͱʹ &YDFMͰه࿥ʜ ఱީσʔλ͸8FC͔Β$47Ͱ μ΢ϯϩʔυ ళฮ*% Ϣʔβ*% Ϩγʔτ*% ߪങ೔࣌ "" BBB  
     ## CCC  
     104σʔλΛ42-Ͱऔಘʜ

24. ͳͥ pandas ͔ • ྫ: খചళͰΩϟϯϖʔϯɾఱީ͕ചΓ্͛ʹ༩͑ΔӨڹΛ஌Γ͍ͨ • ༷ʑͳߟྀࣄ߲… ళฮ*% Ϣʔβ*%

    Ϩγʔτ*% ߪങ೔࣌ "" BBB  
     ## CCC  
     ೔࣌ ৔ॴ ఱީ 
     ౦ژ౎໨ࠇ۠ ੖Ε Ωϟϯϖʔϯ໊শ ։࢝೔ ऴྃ೔ 999   :::   ෼୯ҐͰूܭ͍ͨ͠ ৽ฉ޿ࠂͱళ಄νϥγͰ͸ޮ Ռ͕ҧ͏ʁ Ұఆ࣌ؒ͝ͱͷ؍ଌ஋Λ
    ద౰ͳϧʔϧͰඥ෇͚
    

25. pandas ͷҐஔ෇͚ CRISP-DM: Cross Industry Standard Process for Data Mining

    Ϗδωεͷཧղ σʔλͷཧղ σʔλ४උ ϞσϦϯά ධՁ ల։ ݱ࣮ͷσʔλΛཧղ͠ɺ
    ෼ੳͰ͖Δܗʹམͱ͢

26. pandas ΤίγεςϜ • I/O • pandas-datareader • pandas-msgpack • pandas-gbq

    • υϝΠϯ • geopandas • xarray • ػցֶश • sklearn-pandas • pandas-ml • ฒྻॲཧ • Dask • ։ൃऀ޲͚ • pandas-compat Լઢ͸pandas/pydataϦϙδτϦʹ͋Δ΋ͷ

27. ίϛολͬͯ ͲΜͳ͜ͱΛͯ͠Δʁ

28. pandas ͰͷOSS׆ಈܦҢ 1ϓϧϦΫΤετ=1ίϛοτӡ༻ Number of Commits ॳΊͯͷissue ·ͱ΋ʹ(?)׆ಈ։࢝ ίΞνʔϜՃೖ

29. ίϛολʹͳΔʹ͸ • ϓϩδΣΫτʹΑͬͯҧ͏͕ɺpandasͷ৔߹: • ࣭ɾྔͱ΋े෼ͳߩݙΛ͍ͯ͠Δ͜ͱ • Ұ೥Ҏ্׆ಈ͍ͯ͠Δ͜ͱ →ݱࡏͷίΞνʔϜϝϯόͷਪનɺ౤ථͰܾఆ

30. ίϛολͷ໾ׂ • Make decisions about: • The overall scope, vision

    and direction of the project. • Strategic collaborations with other organizations or individuals. • Specific technical issues, features, bugs and pull requests. • The services that are run by the project. • Regular community discussion doesn’t produce consensus.

31. ίϛολͷۀ຿ • Issue΁ͷճ౴ • ίʔυϨϏϡʔ • ϓϧϦΫΤετͷϚʔδ • पลΠϯϑϥ(CI౳)ͷ੔උɺӡ༻ •

    ϦϦʔε (ϦϦʔεϚωʔδϟ͕࣮ࢪ) • ςϨίϯ

32. ॏཁͳ͜ͱ • ϓϩμΫτͷ඼࣭ • ίϛϡχςΟͷ඼࣭ • ͜ΕΒΛҡ࣋͢ΔͨΊͷ࢓૊Έͮ͘Γ

33. ϓϩμΫτͷ඼࣭ • ࢖͍΍͍͢ • Πϯετʔϧ • ϢʔβϏϦςΟ • υΩϡϝϯτ •

    ඼࣭͕୲อ͞Ε͍ͯΔ • ܧଓతΠϯςάϨʔγϣϯ • ύϑΥʔϚϯεςετ

34. Πϯετʔϧ • Anaconda • ֤छ wheel ͷఏڙ

35. ϢʔβϏϦςΟ • API • ύοέʔδ಺֎Ͱͷ౷Ұੑ • ໊લۭؒΛద੾ʹ෼ׂ(ΞΫηα) • ޙํޓ׵ੑ •

    ௚ͪʹมߋͤͣඇਪ঑Խˠ2ϝδϟʔόʔδϣϯޙʹมߋ • աڈʹγϦΞϥΠζ͞ΕͨΦϒδΣΫτͷಡΈࠐΈΛ୲อ

36. υΩϡϝϯτ • ެࣜυΩϡϝϯτ • http://pandas.pydata.org/pandas-docs/stable/ • ॻ੶ • PythonʹΑΔσʔλ෼ੳೖ໳ /

    Wes McKinney • ༗ࢤυΩϡϝϯτ • Modern pandas / Tom Augspurger • https://tomaugspurger.github.io/modern-1.html

37. ܧଓతΠϯςάϨʔγϣϯ • ੩తνΣοΫ (flake8) • ࣗಈςετ (Travis-CI, AppVeyor, Circle-CI) •

    Python όʔδϣϯ x पล؀ڥͷ૊Έ߹Θͤ • ΧόϨοδνΣοΫ (Codecov)

38. ύϑΥʔϚϯεςετ • Airspeed Velocity • ࢦఆίϛοτͰςετˍίϛοτؒൺֱ • 1000↑ͷϕϯνϚʔΫΛ࣮ߦ http://pandas.pydata.org/speed/pandas

39. ॏཁͳ͜ͱ • ϓϩμΫτͷ඼࣭ • ίϛϡχςΟͷ඼࣭ • ͜ΕΒΛҡ࣋͢ΔͨΊͷ࢓૊Έͮ͘Γ

40. ίϛϡχςΟͷ඼࣭ • ΦʔϓϯͰ͋Δ͜ͱ • ࢀՃ͠΍͍͢ • αϙʔςΟϒͰ͋Δ

41. ࢀՃ͠΍͘͢͢ΔͨΊʹ • Φʔϓϯͳίϛϡχέʔγϣϯ (GitHub, Gitter) • ߦಈنൣ (Code Of Conduct)

    • ։ൃऀ޲͚υΩϡϝϯτ(GitHub Wiki)

42. GitHub • దٓλά෇͚͠ɺٞ࿦ʹࢀՃ͠΍͘͢ • IssueςϯϓϨʔτΛར༻͠ɺॳ৺ऀͰ΋Ϩ ϙʔτ͠΍͘͢

43. ߦಈنൣ (Code of Conduct) • ΨόφϯεͷͨΊͷGitHubϦϙδτϦΛ੔උ • https://github.com/pandas-dev/pandas- governance

44. ։ൃऀ޲͚υΩϡϝϯτ • ։ൃऀ޲͚ • ίϯτϦϏϡʔγϣϯΨΠυ • Gitͷ࢖͍ํ • ςετํ๏ •

    ίʔσΟϯάελΠϧ • ϦϦʔεϊʔτͷॻ͖ํ • Specialities • … • ϝϯςφ޲͚ • ϦϦʔε࣌ͷνΣοΫϦετ • υΩϡϝϯτެ։ํ๏ • …

45. ։ൃऀ޲͚υΩϡϝϯτ • Specialities • ػೳผͷ༗ࣝऀҰཡ(ίΞνʔϜҎ֎΋ؚ) • ؔ࿈͢Δ Issue / ϓϧϦΫΤετ͔Β௨஌Λ

    ඈ͹͢ • https://github.com/pandas-dev/pandas/ wiki/Specialities

46. ஂମɾاۀ͔Βͷࢧԉ • • Institutional Partners • Continuum Analytics • Two

    Sigma

47. OSS׆ಈ ࠷ॳͷҰา

48. OSS׆ಈͷϝϦοτ • ঝೝཉٻ͕ຬͨͤΔ • ༗ࣝऀͷίʔυϨϏϡʔʹΑΓεΩϧΞοϓͰ ͖Δ • ಺෦࣮૷͕ཧղͰ͖ɺޮ཰తͳίʔυ͕ॻ͚Δ • मਖ਼ΛϚελʹऔΓࠐΜͰ΋Β͏͜ͱͰɺݸਓ

    ͷϝϯςෛՙΛܰݮͰ͖Δ

49. OSS׆ಈΛ࢝ΊΔʹ͸ • ίʔυमਖ਼͚͕ͩOSS׆ಈͰ͸ͳ͍ • એ఻͢Δ (ϒϩάΛॻ͘ɺൃද͢Δ) • σʔλܥͷ৔߹ɺOSSΛ࢖ͬͨ෼ੳϊ΢ϋ΢Ͱ΋ • ࣭໰

    (StackOverflow, GitHub Issue)ʹ౴͑Δ • IssueΛॻ͘ • ϓϧϦΫΤετΛૹΔ

50. OSS׆ಈΛ࢝ΊΔʹ͸ • Կʹߩݙ͢Δ͔ΛܾΊΔ • ݸਓϓϩδΣΫτͱͯ͠ • ࣗࣾϓϩμΫτΛOSSͱͯ͠ • ۀ຿Ͱ࢖͏OSSʹରͯ͠ •

    ༗໊ͳOSSʹରͯ͠

51. ϓϧϦΫΤετ͕ ૹΕΔΑ͏ʹͳΓ͍ͨʂ

52. ॳΊͯͷϓϧϦΫΤετ • ·ͣ͸ҰͭͷϓϩδΣΫτ/ػೳ͔Β • ϓϩηε/πʔϧͷशख़ʹ͸ίετ͕͔͔Δ • ίʔυͷؔ܎ΛಡΈղ͘ͷ͸೉͍͠

53. ॳΊͯͷϓϧϦΫΤετ • ؆୯ͳ΋ͷ͔Β • υΩϡϝϯτͷվగ • Τϥʔϝοηʔδͷվળ • қ͍͠όάमਖ਼

54. ϓϧϦΫΤετͷྲྀΕ • ௚͍ͨ͠ issue Λ୳͢ • issueʹͳ͍৔߹ɺ·ͣissueΛॻ͍ͨํ͕ྑ͍ • ίʔυΛमਖ਼͢Δ •

    ϓϧϦΫΤετΛૹΔ

55. ϓϧϦΫΤετର৅ͷ୳͠ํ • େ͖ΊͷϓϩμΫτͷ৔߹ɺλάͰ੔ཧ͞Ε͍ͯ Δ • ෼໺΍೉қ౓Ͱର৅Λ୳͢ • ଞͷϢʔβ͕ணख͍ͯ͠Δ৔߹͸ආ͚ͨํ͕ྑ͍

56. ϓϧϦΫΤετͷૹΓํ • ࠷௿ݶͷϧʔϧΛकΔ • ໌ࣔ͞Ε͍ͯΔ΋ͷ͚ͩͰྑ͍ • ςετΛ௥Ճ͢Δ • ίʔσΟϯάن໿ΛकΔ •

    …

57. ϓϧϦΫΤετΛૹͬͨޙ • ։ൃऀ΍ଞϢʔβʹΑΔϨϏϡʔ • (ϓϩδΣΫτʹΑΓ)υΩϡϝϯτɺϦϦʔεϊʔτͷ੔උ • Ϛʔδ

58. ϋʔυϧʹײ͡ΔཁҼ • ӳޠ • ίϛϡχςΟϩʔΧϧϧʔϧ • Gitͷ࢖͍ํ • ٕज़ྗ •

    Կ͔໰୊͕͋Ε͹ڭ͑ͯ͘ΕΔͷͰɺա౓ʹؾΛ࢖͏ඞཁ͸ͳ͍

59. ϓϧϦΫΤετʹ͸ ͲΜͳ΋ͷ͕͋Δʁ

60. ϓϧϦΫΤετͷछྨ • υΩϡϝϯτमਖ਼ • ෆ۩߹मਖ਼ • ػೳ௥ՃɺAPIมߋ • ϦϑΝΫλϦϯά •

    ςετ௥Ճ • …

61. ϓϧϦΫΤετࣄྫ (1/3) υΩϡϝϯτमਖ਼ • मਖ਼಺༰Λ؆ܿʹهࡌ https://github.com/pandas-dev/pandas/pull/13312 ͜ͷυΩϡϝϯτɺͳΜ͔յΕͯΔͬΆ͍

62. ϓϧϦΫΤετࣄྫ (1/3) υΩϡϝϯτमਖ਼ ۭനߦΛೖΕ͚ͨͩ

63. • ෳࡶͳมߋͷ৔߹͸मਖ਼લޙͷڍಈΛ੔ཧ͢ Δ ࠓͷڍಈ͸
    ʜ
    ͕ͩ
    मਖ਼ޙͷڍಈ͸
    ʜ
    ʹͳΔɻ
    ͜Ε͸ʜ
    ͱ͍͏ϧʔϧʹج͍͍ͮͯͯʜ
    https://github.com/pandas-dev/pandas/pull/13849 ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ

64. • ϨϏϡʔࢦఠΛड͚ͭͭɺ࢓༷ΛݻΊΔ 999ͷ࣌ʹΤϥʔग़ͨ͠ํ͕Α͘ͳ͍ʁ ͦ͏ͳΜ͚ͩͲɺ999Λېࢭ͢Δͱผͷػೳ͕ಈ͔ͳ͘ͳΔ ͡Ό͋࢓ํͳ͍ͶɻଞͷΈΜͳ͸Ͳ͏ࢥ͏ʁ Ҏ߱ٞ࿦ ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ

65. • ϦϦʔεϊʔτ΍υΩϡϝϯτΛमਖ਼͢Δ ϓϧϦΫΤετࣄྫ (2/3) ࢓༷มߋ

66. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • ҙਤ͕఻ΘΒͳ͚Ε͹ίʔυΛࣔ͢ https://github.com/pandas-dev/pandas/pull/13312 ͍΍ɺผͷͱ͜௚ͨ͠ํ͕͍͍Μ͡Όͳ͍ʁ ӳޠ͕͓͔͍͔͠΋ʜɻ
    मਖ਼ՕॴΛϓϧϦΫͰૹͬͨͷͰ֬ೝͯ͠ ࠓͷڍಈ͸
    ʜ
    Ͱ
    

    मਖ਼ޙͷڍಈ͸
    ʜ
    ʹͳΔɻ
    ͨͿΜ
    ʜ
    ͷॲཧΛमਖ਼͢Ε͹͍͍

67. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • (Өڹൣғ͕େ͖͍৔߹)ͻͨ͢ΒςετΛ଍͢ ҟͳΔΫϥε͕ಉ͡ڍಈʹͳ Δ͔νΣοΫ

68. ϓϧϦΫΤετࣄྫ (3/3) όάमਖ਼ • (Өڹൣғ͕େ͖͍৔߹)ͻͨ͢ΒςετΛ଍͢ ܕͷ૊Έ߹ΘͤΛ໢ཏతʹ νΣοΫ ͦΕͧΕͷܕͷ૊Έ߹Θ͕ͤ /VN1Zͱಉ͡ڍಈʹͳΔ͔ νΣοΫ

69. ؾΛ͚͍ͭͯΔ͜ͱ • ίϛϡχέʔγϣϯ • ӳޠ͸௨͡ͳ͍͔΋͠Εͳ͍͕ɺίʔυͳΒ఻ΘΔ • ίʔυमਖ਼ • ہॴతͳमਖ਼Λආ͚Δ •

    ςετΛͪΌΜͱॻ͘ • ࣗ෼Ҏ֎͕ͦͷίʔυΛ৮ͬͨࡍ΋όάͷຒΊࠐΈΛ༧๷Ͱ͖Δ͔ʁ • υΩϡϝϯτ΋Ͱ͖Δ͚ͩॻ͘ • (࢓༷͕͔ͬ͠Γ఻ΘΕ͹)୭͔௚ͯ͘͠ΕΔ

70. ΍ͬͺΓͪΐͬͱ େมͦ͏…ʁ

71. ࠓ೔͔ΒͰ͖Δ͜ͱ • ؾʹͳΔϓϩδΣΫτΛWatchͯ͠ΈΔ • ίϛϡχςΟͷงғؾ͕Θ͔Δ • ໘നͦ͏ͳ Issue / ϓϧϦΫΤετ͕͋Δ͔΋ʁ

72. ·ͱΊ

73. ·ͱΊ • PythonͱՊֶܭࢉ • pandasͰͷOSS׆ಈࣄྫ • OSS׆ಈ ࠷ॳͷҰา • PythonՊֶܭࢉίϛϡχςΟʹࢀՃͯ͠ΈΑ͏

    • ؾʹͳΔϓϩδΣΫτΛWatchͯ͠ΈΑ͏