The Ubiquitous Graph

Big Data eXchange 2014

Transcript

1. The Ubiquitous Graph Use cases from the real world Tareq Abedrabbo - Big Data eXchange 2014
2. About me • CTO at OpenCredo • Working on graph applications for 4 years on a number of different projects • Co-author of Neo4j in Action (Manning)
3. This talk is about the data rather than the technology
4. Use cases: - Impact Analysis - Flow Optimisation
5. Use case 1: Network Impact Analysis
6. Domain: a telco network. Millions of connected network components, services and customers
8. Requirement: Identify the impact of failing components
11. Requirement: Identify interesting patterns, such as single points of failure
13. Observations
14. The network is semi-structured
15. Labelled property graph is a natural fit for the model
16. Additional dimensions can be added to capture abstract concepts: network redundancy, load-balancing
18. Neo4j Cypher queries are a natural solution to the problem
19. Possible evolution: - Multiple starting points - Impact on quality of service - Abstraction of repeatable patterns
20. Why should I care?
21. Variation: Supply Chain Management
23. Use case 2: Oil Flow Optimisation
24. Domain: an oil extraction network. Hundreds of connected components with complex configuration options
26. Requirement: Identify potential configurations to maximise flow
27. Interlude = Genetic Algorithms =
28. Search heuristic that mimics the process of natural selection - Wikipedia
29. Recipe
30. 1. Start from an initial population of candidate solutions 2. Assess each solution using a fitness function 3. Apply genetic operators to derive a new and potentially fitter generation 4. Rinse and repeat!
32. More in detail...
33. 1. Start from an initial population of candidate solutions (individuals or phenotypes) ideally random, diverse and large 2. Attribute a score to each solution using a fitness function (the only place with specific business knowledge) 3. Apply genetic operators to create a new generation - Cross-breeding to retain best characteristics from each parent - Mutation to maintain diversity and to avoid converging to a local optima too quickly
34. Fitness function
36. Crossbreeding
38. Mutation
40. There are other genetic operators - Copy n fittest solutions unchanged - Carry over n unfit candidates - Carry over n randomly chosen candidates
41. Pros: - All domain knowledge is in one place - Explore interesting solutions including counterintuitive ones - Tweak parameters to generate different solutions - Stop when you want
42. Cons: - Fitness function can become really complex and slow - Resulting solutions are not guaranteed to be practical or pretty - Solutions can get worse as the fitness function improves - There is almost always a better solution
43. Observations
44. Simply connected network with complex components
45. Is this even a valid use case for graph database? (hint: yes)
46. Persist and share calculated solutions
47. Inspect intermediary steps
48. Use Cypher queries to interrogate generated solutions
49. Possible evolution: - Identify the most practical and valuable adjustments to the network
50. Why should I care?
51. Variation: Resource Allocation Optimisation
53. To summarise...
54. Graphs are everywhere
55. ...but each graph is different
56. Data-centric vs Domain-centric
57. Graph Domain-centric Data-centric Data model Well-defined Complex Data structure Flexible but predictable Potentially unpredictable Data sources Mostly the application Multiple external sources Design approach Top-down Bottom-up
58. Graphs are data-driven
59. Things to do with a graph: - Traversal and pattern matching: Cypher - Graph algorithms: shortest path, disconnected components, graph saliency, etc... - Optimisation algorithms - Graph analytics
60. Listen to the data, know the domain
61. Think in graphs
62. When designing your data model
63. But also, when testing your graph applications
64. Links OpenCredo: http://www.opencredo.com Neo4j in Action: http://www.manning.com/partner/ Twitter: @tareq_abedrabbo Personal blog: http://www.terminalstate.net Thank you! Any questions?