Software dependency analysis with graph databases (Neo4j)

Slide 1

Slide 1 text

Repo Census Software Dependency Analysis with Graph Databases Philipp Haußleiter & Oliver Tigges

Slide 2

Slide 2 text

Agenda Software Dependency Management Repo Census: Use Cases and Demo Graph Model for Software Dependencies Queries and Cypher

Slide 3

Slide 3 text

Software Dependencies

Slide 4

Slide 4 text

Declaration of Dependencies 4.0.0 com.innoq.census census-webapp 1.0 repo-census-webapp javax.servlet javax.servlet-api 3.1.0 org.springframework org.springframework 4.1.3.RELEASE org.hibernate hibernate-core 4.3.9.Final

Slide 5

Slide 5 text

Transitive Dependencies

Slide 6

Slide 6 text

Complexity explosion

Slide 7

Slide 7 text

Repo Census

Slide 8

Slide 8 text

Use Cases Keep overview and be aware of changes Estimate impacts of software updates License issues Security issues

Slide 9

Slide 9 text

System Architecture Akka (Worker) Play2 (Web UI) Query Result neo4j (GraphDB) Add Artifact Process Artifact Job Request Artifact Job Add Artifact Job Eclipse Aether Client … Process Artifact Request Artifact (Maven) Repository … activeMQ (Queue)

Slide 10

Slide 10 text

Demo Short introduction of the web app Goto to Maven Census

Slide 11

Slide 11 text

Graph Model

Slide 12

Slide 12 text

Transfer to Neo4j Node Elements to handle: Dependencies License Version GroupId ArtifactId

Slide 13

Slide 13 text

Conceptual model com.explodingpixels macwidgets-maven-modul 0.9.6-SNAPSHOT 4.0.0 com.explodingpixels macwidgets-demo macwidgets-demo GNU Lesser General Public License http://www.gnu.org/licenses/lgpl.html com.explodingpixels macwidgets 0.9.6-SNAPSHOT

Slide 14

Slide 14 text

Maven Graph Model

Slide 15

Slide 15 text

Maven Graph Model

Slide 16

Slide 16 text

See model in action Goto to Neo console

Slide 17

Slide 17 text

Cypher Queries

Slide 18

Slide 18 text

Search Find by unique version ID MATCH (v:Version {uId: 'com.innoq.census:census-webapp:jar:1.0'}) RETURN v.uId as UID, v:ToDo AS todo, v:Processing AS processing Search with regular expression MATCH (v:Version) WHERE v.uId =~ 'com.innoq.*' RETURN v.uId as UID

Slide 19

Slide 19 text

Get license info MATCH (v:Version {uId: {0}})-[:HAS_LICENSE]->(l:LicenseInfo) RETURN l.name AS lic_name, l.url AS url ORDER BY lic_name

Slide 20

Slide 20 text

Show other versions MATCH (v:Version)-[:VERSION_OF]->(ae:ArtifactElement) <-[:VERSION_OF]-(o:Version) WHERE v.uId = {0} RETURN o.uId AS id ORDER BY id

Slide 21

Slide 21 text

Show direct dependencies MATCH (v:Version {uId: {0}}) OPTIONAL MATCH v-[dr:DEPENDS_ON]->other RETURN other.uId AS dep, dr.scope AS scope ORDER BY dep

Slide 22

Slide 22 text

Show transitive dependencies MATCH (v:Version) WHERE v.uId =~ 'com.innoq.*1.0' OPTIONAL MATCH (v)-[r:DEPENDS_ON_COMPILE|DEPENDS_ON_RUNTIME*1..10]->other WITH distinct other as deps RETURN deps.uId

Slide 23

Slide 23 text

Transitive dependency paths MATCH (v:Version) WHERE v.uId =~ 'com.innoq.*1.0' OPTIONAL MATCH p = (v-[r:DEPENDS_ON_COMPILE|DEPENDS_ON_RUNTIME*1..10]->oth WITH tail ( extract (d IN nodes(p) | d.uId) ) as deps RETURN deps

Slide 24

Slide 24 text

License overview of all deps MATCH (v:Version) WHERE v.uId =~ 'com.innoq.*1.0' OPTIONAL MATCH v-[r:DEPENDS_ON_COMPILE|DEPENDS_ON_RUNTIME*1..5]->other -[:HAS_LICENSE]->(l:LicenseInfo) RETURN DISTINCT l.name as License, l.url as URL ORDER BY l.name

Slide 25

Slide 25 text

Changes between versions MATCH (orig:Version {uId: {0}}), (other:Version {uId: {1}}) OPTIONAL MATCH (orig)-[:DEPENDS_ON]->(common:Version)<-[:DEPENDS_ON]-(other) WITH orig, other, collect(DISTINCT common.uId) AS common OPTIONAL MATCH (orig)-[:DEPENDS_ON]->(origVersion:Version)-[:VERSION_OF]->(commonArtifact:ArtifactElement), (other)-[:DEPENDS_ON]->(otherVersion:Version)-[:VERSION_OF]->(commonArtifact) WITH orig, other, common, collect({origVersion: origVersion.uId, otherVersion: otherVersion.uId}) AS commonArtifacts OPTIONAL MATCH (orig)-[:DEPENDS_ON]->(origOnly:Version)-[:VERSION_OF]->(a:ArtifactElement) WHERE NOT (other)-[:DEPENDS_ON]->(:Version)-[:VERSION_OF]->(a) WITH orig, other, common, commonArtifacts, collect(DISTINCT origOnly.uId) AS origOnly OPTIONAL MATCH (other)-[:DEPENDS_ON]->(otherOnly:Version)-[:VERSION_OF]->(a:ArtifactElement) WHERE NOT (orig)-[:DEPENDS_ON]->(:Version)-[:VERSION_OF]->(a) RETURN common, commonArtifacts, origOnly, collect(DISTINCT otherOnly.uId) AS otherOnly

Slide 26

Slide 26 text

Changes between versions

Slide 27

Slide 27 text

Project Status

Slide 28

Slide 28 text

Limits Early Stage Quantity structures Limits of the System

Slide 29

Slide 29 text

Outlook

Slide 30

Slide 30 text

Planned features Support for Multiple Repositories Additional Attributes: Organization, Developers, Properties, Repositories, Plugins Support for (Multi-)User Project Upload (Pom) Social Login (Github/Twitter/...)

Slide 31

Slide 31 text

Integration & Cooperation Support other Repositories: Gems, RPMs, NPMs, ... Integration in build process Integration of jQAssistant

Slide 32

Slide 32 text

Availability Project Page How to get, test and contribute SaaS Appliance (OVA)

Slide 33

Slide 33 text

Q&A

Slide 34

Slide 34 text

Thank you! Philipp Haußleiter philipp.haussleiter@innoq.com @phaus Oliver Tigges oliver.tigges@innoq.com @otigges