Neo4J Tales from the Trenches: A Recommendation Engine Case Study

Slides accompanying talk done with Michal Bachman at SkillsMatter. Video available here
Part 1: https://skillsmatter.com/skillscasts/3206-neo4j-tales
Part 2: https://skillsmatter.com/skillscasts/3341-neo4j-tales-pt2

Transcript

1. Neo4J – Tales from the Trenches A RECOMMENDATION ENGINE CASE STUDY Michal Bachman & Nicki Watt @bachmanm & @techiewatt
2. Who we are … Nicki Watt Michal Bachman OpenCredo works for works for colleague of role = “consultant” role = “consultant” Neo4J Opigram partner of uses works on works on
3. Opigram •  Opinion Profile •  Recommendation/Insight Engine •  Social Network (Future) •  http://labs.yougov.co.uk
4. Things Opinions Panelists about provides Recommendations/ Interesting Insights People who like … also tend to like … People who like … tend to support … People who like … describe themselves as …. generates About (themselves) Opigram
5. Problem Scale •  ~ 150k panelists (a.k.a. users) •  ~ 100k “things” (movies, books,…) •  ~ 8M relationships •  All that UK only •  Soon to be launched in the US
6. Live Demo
7. Opigram •  Started Feb 2011 •  Nov 2011 – OpenCredo – Many lessons learned
8. Neo4J •  Graph Database •  Schema-less (NoSQL) •  Vertices and Edges •  a.k.a. Nodes and Relationships •  Traversals
9. Neo4J Nicki Watt Michal Bachman OpenCredo works for works for colleague of role = “consultant” role = “consultant” Neo4J Opigram partner of uses works on works on
10. Opigram + Neo4J •  Taxonomy of “things” •  Opinions on “things” •  Recommendations •  Offline “Crunching”
11. Opigram + MySQL •  CMS Functionality •  Crunching Results •  Configuration / Metadata
12. Lessons Learned •  Everyone loves Neo4J! Find praise online •  “Trenches Talk” – insight into some real challenges – approaches to solutions •  We have 5 practical lessons for you – Tips – Tricks – Troubles
13. Lessons Learned •  Lesson 1: Graph “Schema” •  Lesson 2: Neo Node IDs •  Lesson 3: Graph-wide Operations •  Lesson 4: Extracting Randomised Data •  Lesson 5: Multi-threading
14. Lesson 1 Graph “Schema”
15. Schema-less ≠ Credit: Greencolander
16. Michal Pulp Fiction review text = “…” Movie type Descriptor Boring Cool Funny Romantic type type type type described as described as votes = 1 votes = 1 descriptors = Cool, Funny
17. Michal Pulp Fiction review of Movie type Descriptor Boring Cool Funny Romantic type type type type described as described as Review text=“…” created
18. Lesson 1: Conclusion •  Think about your graph structure •  Tailor for questions asked •  Important things => nodes •  Evolve your graph •  More about data migration in Lesson 3
19. Lesson 2 Neo4J Node IDs
20. Neo Node IDs •  What are they •  How can I use them in my application – You should not! •  Why not – Not Stable – Ids are recycled over time, only guaranteed to be unique during a specific time span – Internal Neo implementation detail
21. Example “User Transformation”
22. Michal Descriptor Boring Funny Romantic type type type Nicki Jim MySQL USER_ID NEO_ NODE_ID ACTIVE 101 1 Y 102 2 Y 103 3 Y Panelist type type type 1 2 4 5 6 7 8 USER_ID NEO_ NODE_ID ACTIVE 101 1 Y 102 2 Y 103 3 N Cool type 3 Jim has now become Cool !
23. Alternate ID Strategies •  Challenge: Find a starting point in graph •  Client provided IDs – Add as a standard property on the node – Add to index (or use auto indexer) •  Natural vs. Synthetic IDs •  Auto generate your own IDs – Hook into Neo4J Transaction Kernel – Use auto indexer
24. Auto generate your own IDs 1)  Implement TransactionEventHandler 2) Register TransactionEventHandler with graphDatabaseService 3) Turn auto indexing on for seamless generation
25. Lesson 2: Conclusion Neo4J Node IDs Don’t expose! + Index is your friend •  Use an index to lookup specific nodes •  If used inappropriately may result in inaccurate data & unexpected behaviour •  Internal Neo implementation detail – subject to change
26. Lesson 3 Graph-wide Operations
27. Motivations •  Fixes – Bugs – Re-indexing •  “Schema” Migrations •  Data Export •  Data Analysis •  Count Caching
28. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete relationships only from one side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
29. Example Deleting “soft-deleted” relationships
30. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete only from 1 side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
32. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete only from 1 side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
34. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete only from 1 side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
36. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete only from 1 side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
37. Example Computing statistics
39. Lesson 3: Graph-wide Operations •  Batch Updates •  Delete only from 1 side •  GlobalGraphOperations since 1.6 •  No need for TX when reading
40. Lesson 4 Extracting Randomised Data
41. Extracting Randomised Data •  Use Cases – Provide Random Suggestions to users – Use for statistical analysis aka “Random Sampling” •  Problem – No built in Neo4J support – Not Neo4J’s sweet spot – May result in very bad performance
42. Options •  Randomisation Strategies – “Load, Shuffle, Pick” – “Hit and Miss” – Custom Relationship Expander/Evaluator – Reservoir Sampling •  Performance Helpers – Indexes – Front with a cache if need be
43. Custom Relationship Evaluator
44. Reservoir Sampling Algorithm ** With custom short circuit option
45. Traversals vs. Index 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 5000 10000 20000 40000 80000 160000 1.5 TRAVERSAL PASS 1 (COLD) 1.4.2 TRAVERSAL PASS 1 (COLD) 1.4.2 TRAVERSAL PASS 2 (WARMISH) 1.5 TRAVERSAL PASS 2 (WARMISH) 1.5 INDEX 1.6.2 TRAVERSAL PASS 1 (COLD)" 1.6.2 TRAVERSAL PASS 2 (WARMISH) 25 random nodes extracted from [Sample Size] using “Reservoir Sampling” algorithm X-Axis: Sample Size Y-Axis: Time (milliseconds) Use of indexes reduced time to +- 300 - 1000ms from cold
46. Conclusion •  Most options are not “truly random” more “randomish” •  Primarily has bad performance when hitting cold parts of graph ** This is generally true of any persistence technology which needs to perform random scattered disk access •  Performance can be improved with – Indexes – Caching
47. Lesson 5 Multi-threading
50. Lesson 5: Multi-threading •  Shortcoming in Neo4J •  Fixed in version 1.7 •  Avoid relationship properties in multi- threaded pre-1.7 apps
51. Questions?
52. Beer Time! •  @bachmanm •  michal.bachman@opencredo.com •  @techiewatt •  nicki.watt@opencredo.com