Upgrade to Pro — share decks privately, control downloads, hide ads and more …

There and Back Again: A Developer's Tale

There and Back Again: A Developer's Tale

Every developer has had to learn a new technology from hearing about it to research to an end solution. It's not always easy to get started with any new technology and make decisions on import, access, integration, and implementation. Graph provides new capabilities many have not leveraged yet and many developers want to apply it to a need in their companies. Graph is different from other data storage solutions that exist today, so how can we learn about and evaluate decisions to apply this technology in the right way?
In this presentation, we see how to go from no knowledge to an end-to-end solution with Neo4j. Using a data set of the Lord of the Rings movie network, you can learn how to apply each of the process steps and what the result looks like. From learning the model and structure, choosing which environment install, discovering various possibilities for importing data, to actually loading, querying, and presenting the value to others, we show how to make decisions along the way and resources to use in making those decisions. Go from 0 to 60 in building your next solution with a graph database!


Jennifer Reif

June 25, 2019


  1. There And Back Again… A Developer’s Tale By: Jennifer Reif

    Neo4j Developer Relations Engineer jennifer.reif@neo4j.com @JMHReif
  2. Who Am I? • Developer Relations Engineer for Neo4j •

    Continuous learner, developer, blogger • Conference speaker • Survivor of financial industry development Email: jennifer.reif@neo4j.com Twitter: @JMHReif
  3. We have a problem…

  4. We want to know… • What actors played which characters

    in Lord of the Rings movies • Other scenarios: • What employees have which skills for job openings in the company • What customers purchased which products and the suppliers • What patient was prescribed which medications from which doctors • What customers bought which vehicles from what dealerships/people
  5. Existing solutions are painful • Thousands of actors, employees, customers,

    patients, doctors, dealerships, skills, etc • Relational: • Great for reports and simple JOINs, but too many JOINs to go across 3 core tables and lookup tables with endless rows each • Document: • Great for pulling information about individual components, but linking properties across substructures is complicated • Key-value: • Great for bits of information very quickly, but aggregating and compiling lots of related data is arduous
  6. What is a Graph Database?

  7. Graph Chart

  8. Database - specifically graph • Database: a structured set of

    data held in a computer, especially one that is accessible in various ways. • Relational? NoSQL? Graph? • Graph database: uses graph structures for semantic queries with nodes, edges, and properties to represent and store data.
  9. What can graph do?

  10. –Wikipedia, “Graph Database”, Performance section “Execution of queries within a

    graph database is localized to a portion of the graph. It does not search through irrelevant data, making it advantageous for real-time big data analytical queries. Consequently, graph database performance is proportional to the size of the data needed to be traversed, staying relatively constant despite the growth of data stored.”
  11. Relational Graph

  12. Other NoSQL Graph

  13. None
  14. What is it used to accomplish? Use Cases • Social

    networks • Impact analysis • Logistics and routing • Recommendations • Access control • Fraud analysis • …and many, many more!
  15. Lots of graph options… Why choose Neo4j?

  16. Neo4j is a database Neo4j Fast Reliable No size limit

    Binary & HTTP protocol ACID transactions 2-4 M
 ops/s per core Clustering scale & availability Official Drivers
  17. Neo4j is a graph database Neo4j Property Graph Model Native

    GraphDB Schema Free Graph Storage Cypher Query Language Developer Workbench Extensible Procedures & Functions Graph Visualization
  18. What’s the data model?

  19. Whiteboard Friendliness Easy to design and model direct representation of

    the model
  20. Property Graph Data Model • 2 Main Components: • Nodes

    • Relationships • Additional Components: • Labels • Properties
  21. Property Graph Data Model • Nodes: • Represent the objects

    in the graph • Can be categorized using Labels Car Person Person
  22. Property Graph Data Model • Nodes: • Represent the objects

    in the graph • Can be categorized using Labels • Relationships: • Relate nodes by type and direction Car DRIVES LOVES LOVES LIVES WITH OW NS Person Person
  23. Property Graph Data Model • Nodes: • Represent the objects

    in the graph • Can be categorized using Labels • Relationships: • Relate nodes by type and direction • Properties: • Name-value pairs that can be applied to nodes or relationships Car DRIVES LOVES LOVES LIVES WITH OW NS Person Person name: “Dan” born: May 29, 1970 twitter: “@dan” name: “Ann” born: Dec 5, 1975 since: Jan 10, 2011 brand: “Volvo” model: “V70”
  24. Tools for data modeling… • Arrows tool: • http://www.apcjones.com/arrows/ •

    Developer guides: • https://neo4j.com/developer/data-modeling/ • GraphGists: • https://neo4j.com/graphgists/ • Community Site: • https://community.neo4j.com/ • Training - Data Modeling course: • https://neo4j.com/graphacademy/
  25. Applied to our scenario

  26. Whiteboard friendliness

  27. Whiteboard friendliness title: The Lord of the Rings… released: 2003

    Movie Cast name: Orlando Bloom name: Frodo Baggins Character PLAYED APPEARS_IN name: Elijah Wood Cast Character name: Legolas Character name: Aragorn name: Viggo Mortensen Cast PLAYED PLAYED APPEARS_IN APPEARS_IN
  28. Whiteboard friendliness

  29. Importing Data

  30. Options for Importing Data • Cypher statements / script: create

    individual statements to load data manually • LOAD CSV: used for small and medium data sets can import local or online csv files to graph • ETL Tool: can import from a relational database and maps relational data model to graph • Kettle: can import massive amounts of data from a variety of sources • APOC: standard library that includes several import procedures for different data formats • Neo4j-admin import tool: command-line interface for large amounts of data • Import programmatically from drivers: interact via preferred programming language
  31. Cypher Query Language…. SQL for graphs

  32. Tools for Cypher… • Cypher quick-reference: • https://neo4j.com/docs/cypher-refcard/current/ • Developer

    guides: • https://neo4j.com/developer/cypher/ • Cypher manual: • https://neo4j.com/docs/cypher-manual/current/ • Community Site: • https://community.neo4j.com/ • Resources list: • https://neo4j.com/developer/cypher-resources/
  33. Cypher: Powerful and Expressive CREATE (:Person { name:“Dan”}) -[:LOVES]-> (:Person

  34. Cypher: Powerful and Expressive LOVES Dan Ann MATCH (:Person {

    name:"Dan"} ) -[:LOVES]-> ( whom ) 
 RETURN whom
  35. Cypher in 20 sec… • Nodes look like this: •

    (var:Label) OR (var:Label { propKey: propValue }) • Relationships look like this: • -[var:REL_TYPE]-> or -[var:REL_TYPE { propKey: propValue }]- • Using Cypher is just looking for particular patterns of those nodes/rels • (var1:Label)-[var2:REL_TYPE]->(var3:Label)
  36. How do we want to import our data?

  37. Cypher statements/script MERGE (m:Movie {id: 100}) ON CREATE SET m.title

    = “The Lord of the Rings: The Fellowship of the Ring”, m.releaseDate = date(‘2001-12-19’)… MERGE (c:Character {id: 300}) ON CREATE SET m.name = “Legolas”… MERGE (c)-[:APPEARED_IN]->(m) ….
  38. LOAD CSV LOAD CSV WITH HEADERS FROM “file:///movies.csv” as row

    MERGE (m:Movie {id: row.movieId}) ON CREATE SET m.title = row.title, m.releaseDate = date(row.released)… …. LOAD CSV WITH HEADERS FROM “file:///movieCharacters.csv” as row MATCH (m:Movie {id: row.movieId}) WITH m, row MERGE (c:Character {id: row.id}) ON CREATE SET m.name = row.name … MERGE (c)-[:APPEARED_IN]->(m) ….
  39. ETL Tool

  40. Kettle

  41. APOC WITH "https://bestmovies.com/" as url CALL apoc.load.json(url) YIELD value UNWIND

    value.results AS results WITH results MERGE (m:Movie {id: results.id}) ON CREATE SET m.title = results.title, m.releaseDate = date(results.released)… ….
  42. APOC fave procs • apoc.load.json(url) / apoc.load.csv(file) / apoc.load.xml(file) /

    apoc.load.jdbc(url) • Procedures to load various kinds of data • Can handle flat files or url paths (locally or remote) • Excellent when you need transformations with data load • apoc.periodic.iterate(‘cypher1’, ’cypher2’, {parms}) • For each result in cypher1 statement, run cypher2 statement on them • Helpful for selecting a segment for update • apoc.do.when(condition, query, else, {parms}) • Handles transformation for substituting values • Used for a variety of functions, but here is good for cleaning data • apoc.date.format(dateType, “precision”, ‘format’) • Can output date in a variety of formats for display or querying • Very helpful pulling or pushing date/time value into/out of Neo4j
  43. Tools for APOC… • Docs: • https://neo4j-contrib.github.io/neo4j-apoc- procedures/ • Developer

    guides: • https://neo4j.com/developer/neo4j-apoc/ • Community Site: • https://community.neo4j.com/ • YouTube videos: • https://www.youtube.com/watch? v=V1DTBjetIfk&list=PL9Hl4pk2FsvXEww23 lDX_owoKoqqBQpdq
  44. Getting an instance of Neo4j

  45. Free Tools for Running Neo4j… • Sandbox: • https://neo4j.com/sandbox-v2/ •

    Neo4j Desktop (local instance): • https://neo4j.com/download/ • Server install (open source): • https://neo4j.com/download-center/#community • In the Cloud: • https://neo4j.com/developer/guide-cloud- deployment/ • Docker: • https://hub.docker.com/_/neo4j
  46. Getting our data imported…

  47. //Load Movie objects that are wanted WITH 'https://api.themoviedb.org/3/search/movie?api_key='+ $apiKey+'&query=Lord%20of%20the%20Rings' as

    url CALL apoc.load.json(url) YIELD value UNWIND value.results AS results WITH results MERGE (m:Movie {movieId: results.id})
 ON CREATE SET m.title = results.title, m.desc = results.overview, m.poster = results.poster_path, m.reviewStars = results.vote_average, m.reviews = results.vote_count WITH results, m CALL apoc.do.when(results.release_date = "", 'SET m.releaseDate = null', 'SET m.releaseDate = date(results.release_date)', {m:m, results:results}) YIELD value RETURN m
  48. //For Movie objects just loaded, pick out trilogy and retrieve

    cast of those movies WITH 'https://api.themoviedb.org/3/movie/' as prefix, '/credits?api_key='+$apiKey as suffix, ["The Lord of the Rings: The Fellowship of the Ring", "The Lord of the Rings: The Two Towers", "The Lord of the Rings: The Return of the King"] as movies CALL apoc.periodic.iterate('MATCH (m:Movie) WHERE m.title IN $movies RETURN m', 'WITH m CALL apoc.load.json($prefix+m.movieId+$suffix) YIELD value UNWIND value.cast AS cast MERGE (c:Cast {id: cast.id}) ON CREATE SET c.name = cast.name MERGE (ch:Character {name: cast.character}) MERGE (ch)-[r:APPEARS_IN]->(m) MERGE (c)-[r1:PLAYED]->(ch)', {batchSize: 1, iterateList:false, params:{movies:movies, prefix:prefix, suffix:suffix}});
  49. Writing Queries

  50. Let’s Query Our Data!

  51. Other ways to query and explore • Make calls from

    an application • Neo4j drivers for almost any programming language • Java, Python, Javascript, Go, Ruby, PHP • Visualization tools • Open source and proprietary • Neovis, Browser, Bloom, 3d-force-graph, Kineviz, yWorks
  52. Fitting into Architecture

  53. Will it play nice? • Integrations, integrations, integrations! • Out-of-the-box

    plugins (APOC, GraphQL, graph algorithms) • Custom extensions possible • Tons of options for feeding data to existing tools/systems • Tableau, Kettle, Kafka, ElasticSearch, other DBs, Spark, and many more
  54. Sharing the value

  55. What can I show to others? • Neo4j Bloom (or

    partner/open source visualization tools) • Exploration tool for business users to query with natural language • Basic reports and query performance • Build according to specs and compare solutions, just as you would with any technology evaluation • Use cases and success stories • https://neo4j.com/resources • Possible integrations and minimal interruption of existing systems • What tools are you using today? Does our integration fit neatly? • Community and support network! • Support agreement or fabulous expert community answers to questions
  56. Recap! @JMHReif jennifer.reif@neo4j.com