Maven central repository の artifact をランキングする #渋谷java

by KOMIYA Atsushi

Slide 1

Slide 1 text

Maven central repository ͷ artifact ΛϥϯΩϯά͢Δ ौ୩Java #20 2017-09-30 KOMIYA Atsushi

Slide 2

Slide 2 text

@komiya_atsushi

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Today’s topic

Slide 5

Slide 5 text

ΈΜͳ ͍͖ͩ͢ Maven central repository ͷ artifact ΛϥϯΩϯάͯ͠ΈΔ

Slide 6

Slide 6 text

Artifact ΛϥϯΩϯά͢Δ • Maven central repository ্ͷ artifact ͸  (όʔδϣϯҧ͍Λআ͍ͯ) 20 ສҎ্ଘࡏ͢Δ • ΞϓϦέʔγϣϯʹ૊ΈࠐΉϥΠϒϥϦΛબఆ  ͢Δࡍ͸ɺར༻࣮੷ͷ͋ΔϥΠϒϥϦΛબͼ͍ͨ • ௕͍΋ͷʹ͸ר͔Εͯੜ͖͍͖͍ͯͨੑ෼ • Artifact ʹର͢ΔϥϯΩϯά͕ཉ͍͠ʂ

Slide 7

Slide 7 text

http://search.maven.org/#stats

Slide 8

Slide 8 text

Ͳ͏΍ͬͯ artifact Λ ϥϯΩϯά͢Δͷ͔ʁ

Slide 9

Slide 9 text

Artifact ͷґଘؔ܎ʹண໨͢Δ • ʮ͞·͟·ͳ artifact ʹґଘ͞Ε͍ͯΔ artifact ΄ Ͳɺॏཁͳ artifact Ͱ͋Δʯͱ͍͏ԾઆΛߟ͑Δ • Artifact ͝ͱͷʮඃࢀর਺ʯ Λࢦඪ஋ͱ͢Δํ๏͕ ߟ͑ΒΕΔ • ࿦จͷʮඃҾ༻਺ʯతͳߟ͑ํ • ୯७ͳʮඃࢀর਺ʯΑΓΑ͍ࢦඪ஋͸ͳ͍͔ʁ

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

ґଘؔ܎͸ ༗޲άϥϑͰ දݱͰ͖Δ

Slide 12

Slide 12 text

PageRank !

Slide 13

Slide 13 text

PageRank • Google ͷΞϨ • Web ϖʔδͷϦϯΫؔ܎͔Βϖʔδͷॏཁ౓Λଌఆ • ϦϯΫؔ܎͸༗޲άϥϑͱͯ͠දݱͰ͖Δ • Artifact ಉ࢜ͷґଘؔ܎Λ༗޲άϥϑͰදݱ͢Δ • ϊʔυ: artifactɺΤοδ: ґଘؔ܎ • Τοδ͸ґଘ͍ͯ͠Δ → ґଘ͞Ε͍ͯΔɺͷ޲͖ • είΞ͕ߴ͍ artifact ΄ͲɺॏཁͰ͋ΔͱղऍͰ͖Δ

Slide 14

Slide 14 text

Maven central repository ͷ σʔλΛऩू͢Δ

Slide 15

Slide 15 text

Ͳ͏΍ͬͯऩू͢Δʁ • https://repo1.maven.org/maven2/ ͔Βɺ͢΂ͯͷ POM ϑΝΠϧΛͻͨ͢ΒΫϩʔϧ͢Δʁ • όʔδϣϯҧ͍ࠐΈͰ artifact ૯਺͸ 200 ສҎ্… • ࠷৽όʔδϣϯͷ POM ͚ͩμ΢ϯϩʔυ͍ͨ͠ • ͔͠͠ɺͲͷ artifact ͕࠷৽όʔδϣϯͳͷ͔Λ  (จࣈྻͷ) όʔδϣϯ৘ใ͔Β൑ఆ͢Δͷ͸໘౗

Slide 16

Slide 16 text

Index ϑΝΠϧΛར༻͢Δ • ࣮͸ central repository ্ͷ͢΂ͯͷόʔδϣϯͷ artifact ΛؚΜͩ index ϑΝΠϧ͕ఏڙ͞Ε͍ͯΔ • https://maven.apache.org/repository/central- index.html • .properties ϑΝΠϧͱ gzip ѹॖ͞ΕͨϑΝΠϧ  (300 MB ௒) ͷೋͭͰߏ੒͞Ε͍ͯΔ • Weekly Ͱߋ৽͞Ε͍ͯΔ

Slide 17

Slide 17 text

Index ϑΝΠϧͰಘΒΕΔ / ಘΒΕͳ͍৘ใ • Index ϑΝΠϧ͔ΒಘΒΕΔ৘ใ (Ұ෦) • Group ID • Artifact ID • όʔδϣϯ • Classiﬁer (sources / javadoc / linux-x86_64 ͱ͔ͷΞϨ) • Artifact ͷϑΝΠϧͷ࠷ऴߋ৽೔࣌ • ͜ΕͰ࠷৽όʔδϣϯͷ artifact Λ೺ѲͰ͖Δ͸ͣ • Index ϑΝΠϧ͔Β͸ಘΒΕͳ͍৘ใ • Artifact ಉ࢜ͷґଘؔ܎

Slide 18

Slide 18 text

Index ϑΝΠϧͷ૸ࠪ • indexer-reader Λར༻͢Δ • group: 'org.apache.maven.indexer' • name: 'indexer-reader' • ۩ମతͳར༻ํ๏͸ҎԼ URL ͷ࣮૷Λࢀর • http://bit.ly/maven-indexer-demo

Slide 19

Slide 19 text

Artifact ಉ࢜ͷґଘؔ܎ • Maven central repository ্ͷ POM ϑΝΠϧΛ  ࢀর͢ΔҎ֎ʹख͕ͳ͍ͬΆ͍ • ࢓ํͳ͍ͷͰɺͻͨ͢ΒྗٕͰΫϩʔϧ • ֤ Artifact ͷ࠷৽όʔδϣϯʹݶఆ͢Ε͹ɺ  ଟগ͸ϚγʹͳΔ • ͦΕͰ΋ 20 ສҎ্͚ͩͲ…

Slide 20

Slide 20 text

POM ϑΝΠϧͷಡΈࠐΈ • maven-model Λར༻͢Δ • group: 'org.apache.maven' • name: 'maven-model' public static void demo() throws Exception { try (InputStream in = new FileInputStream("path/to/pom.xml")) { Model model = new MavenXpp3Reader().read(in); // ґଘؔ܎͕औಘͰ͖Δ List dependencies = model.getDependencies(); } }

Slide 21

Slide 21 text

PageRank Λܭࢉ͢Δ

Slide 22

Slide 22 text

ࣗલ࣮૷ʁ ൱ʂ

Slide 23

Slide 23 text

Apache Spark / GraphX Λ࢖͏ • GraphX • Spark ্ͰάϥϑΛѻ͍ɺܭࢉ͢ΔͨΊͷ API Λఏڙ͢Δ • PageRank ͕͠Εͬͱ࣮૷͞Ε͍ͯΔ ❤ • άϥϑͷن໛తʹɺLocal mode ͰܭࢉՄೳ

Slide 24

Slide 24 text

Apache Spark / GraphX Λ࢖͏ def run(sc: SparkContext): Unit = { // ਺஋දݱ͞Εͨ 2 ͭͷ artifact Λεϖʔε۠੾ΓͰฒ΂ͯґଘؔ܎Λදͨ͠ϑΝΠϧ val graph = GraphLoader.edgeListFile(sc, "path/to/dependency-graph.txt") // PageRank Λܭࢉ͢Δ val ranking = graph.pageRank(0.0001).vertices // Artifact ͷ਺஋දݱ͔Β GAV (groupId|artifactId|version) ΁ͷϚοϐϯά val artifacts = sc.textFile("path/to/artifacts.txt").map { line => val values = line.split(",") (values(0).toLong, values(1)) } // Artifact ͷ਺஋දݱΛ GAV ʹஔ͖׵͑ͯϑΝΠϧʹॻ͖ग़͢ artifacts.join(ranking).map { case (id, (gav, rank)) => (gav, rank) } .sortBy(_._2, ascending = false) .map(t => t._1 + "," + t._2) .saveAsTextFile("path/to/result") }

Slide 25

Slide 25 text

ґଘؔ܎ͷάϥϑ • Maven ͷґଘؔ܎ʹ͸ʮείʔϓʯ͕͋Δ • compile, provided, runtime, test, system, import • ҎԼͷείʔϓ (ͷ૊Έ߹Θͤ) ͝ͱʹ PageRank Λܭࢉ͢Δ • ͢΂ͯ • compile • test • ͢΂ͯ (ґଘ͞Ε͍ͯΔ → ґଘ͍ͯ͠Δɺͷٯ޲͖)

Slide 26

Slide 26 text

࣮ࡍʹϥϯΩϯάΛݟͯΈΑ͏

Slide 27

Slide 27 text

ϥϯΩϯά݁Ռʹ͍ͭͯ • Top 10 ΋͘͠͸ Top 20 ʹߜͬͯ͝঺հ • Top 100 ·Ͱͷ݁Ռ͸ҎԼʹܝࡌ  (Google εϓϨουγʔτ) • http://bit.ly/PackageRank

Slide 28

Slide 28 text

ϥϯΩϯά: ͢΂ͯ

Slide 29

Slide 29 text

ϥϯΩϯά: ͢΂ͯ (#1~10) 1BHF3BOL HSPVQ BSUJGBDU WFSTJPO KVOJU KVOJU PSHTDBMBMBOH TDBMBDPNQJMFS PSHTMGK TMGKBQJ BMQIB PSHNPDLJUP NPDLJUPDPSF PSHUFTUOH UFTUOH PSHTDBMBUFTU TDBMBUFTU@ PSHNPDLJUP NPDLJUPBMM CFUB KBWBYTFSWMFU TFSWMFUBQJ BMQIB DIRPTMPHCBDL MPHCBDLDMBTTJD PSHPCKFOFTJT PCKFOFTJT http://bit.ly/PackageRank

Slide 30

Slide 30 text

ϥϯΩϯά: ͢΂ͯ (#11~20) 1BHF3BOL HSPVQ BSUJGBDU WFSTJPO KBWBYTFSWMFU KBWBYTFSWMFUBQJ PSHBTTFSUK BTTFSUKDPSF MPHK MPHK PSHPTHJ PSHPTHJDPSF PSHTMGK TMGKMPHK BMQIB PSHTDBMBMBOH TDBMBMJCSBSZ OFUCZUFCVEEZ CZUFCVEEZ PSHTDBMBUFTU TDBMBUFTU@ OFUCZUFCVEEZ CZUFCVEEZBHFOU PSHTMGK TMGKTJNQMF BMQIB http://bit.ly/PackageRank

Slide 31

Slide 31 text

ϥϯΩϯάτοϓͷ܏޲ • ςετؔ࿈ • junit, testng, scalatest, assertj, mockito • ݴޠ • Scala (scala-compiler, scala-library) • ϩά • slf4j, logback, log4j (log4j2 ͡Όͳ͍) • ͦͷଞ • objenesis, byte-buddy, servlet-api, org.osgi.core…

Slide 32

Slide 32 text

ϥϯΩϯά: compile

Slide 33

Slide 33 text

ϥϯΩϯά: compile 1BHF3BOL HSPVQ BSUJGBDU WFSTJPO PSHTDBMBMBOH TDBMBMJCSBSZ PSHTMGK TMGKBQJ BMQIB KVOJU KVOJU DPNHPPHMFHVBWB HVBWB PSHBOUMS BOUMSSVOUJNF PSHBOUMS TUSJOHUFNQMBUF DPNHPPHMFDPEFHTPO HTPO PSHKFUCSBJOT BOOPUBUJPOT DPNHPPHMFDPEFpOECVHT KTS PSHKFUCSBJOTLPUMJO LPUMJOTUEMJC http://bit.ly/PackageRank-compile

Slide 34

Slide 34 text

Slide 35

Slide 35 text

ϥϯΩϯά: test

Slide 36

Slide 36 text

ϥϯΩϯά: test 1BHF3BOL HSPVQ BSUJGBDU WFSTJPO KVOJU KVOJU PSHNPDLJUP NPDLJUPDPSF PSHTMGK TMGKBQJ BMQIB PSHUFTUOH UFTUOH PSHTDBMBUFTU TDBMBUFTU@ PSHNPDLJUP NPDLJUPBMM CFUB DIRPTMPHCBDL MPHCBDLDMBTTJD PSHBTTFSUK BTTFSUKDPSF PSHTMGK TMGKMPHK BMQIB PSHTQPDLGSBNFXPSL TQPDLDPSF HSPPWZ http://bit.ly/PackageRank-test

Slide 37

Slide 37 text

ϥϯΩϯά: ͢΂ͯ (ٯ޲͖)

Slide 38

Slide 38 text

ϥϯΩϯά: ͢΂ͯ (ٯ޲͖) 1BHF3BOL HSPVQ BSUJGBDU WFSTJPO PSHBQBDIFDMFSF[[B QMBUGPSNMBVODIFSTUPSBHFMFT TQBSFOU JODVCBUJOH PSHRJKMJCSBSZ PSHRJKMJCSBSZTIJSPXFC DPNHJUIVCMJWFTFOTF PSHMJWF4FOTFBTTFNCMJFT PSHBQBDIFQPMZHFOFMJCSBSJFT PSHBQBDIFQPMZHFOFMJCSBSZ TIJSPXFC DPNHJUIVCTOPXESFBNBOESPJE XJEHFU PSHCMVFTUFNTPGUXBSFPQFOFPBFYBN QMFBQQMJDBUJPOTQSJOH PSEFSNBOBHFSBQQMJDBUJPO PSHCMVFTUFNTPGUXBSFPQFOFPBFYBN QMFBQQMJDBUJPOTQSJOH XBSFIPVTFNBOBHFS BQQMJDBUJPO LSQFLXPOOBNTQZNFNDBDIFEFYUSB USBOTDPEFST TQZNFNDBDIFEFYUSB USBOTDPEFSTDPSF PSHBQBDIFTFSWJDFNJYCVOEMFT PSHBQBDIFTFSWJDFNJYCVOEM FTBXTKBWBTEL @ NFUBUBSLBHTPOWBMVF HTPOWBMVF http://bit.ly/PackageRank-inverted

Slide 39

Slide 39 text

·ͱΊ

Slide 40

Slide 40 text

·ͱΊ • Artifact ͷґଘؔ܎Λ΋ͱʹ PageRank Λܭࢉ ͠ɺartifact ΛϥϯΩϯάͯ͠Έͨ • ·͋·͋ଥ౰ͳ݁Ռ…͔ͳʁ • ࠷ۙެ։͞Εͨɺྺ࢙ͷઙ͍ artifact ͚ͩʹߜͬ ͯ PageRank Λܭࢉͯ͠Έ͍ͨ • ࠷ۙͷτϨϯυతͳ artifact Λݟ͚ͭΔ͜ͱ͕ Ͱ͖Δ͔΋

Slide 41

Slide 41 text

Thank you!