Titan: The Journey

Lessons learned while deploying Titan to a production environment.

A few things to keep in mind:

* Everything said here should be taken with a grain of salt

* Our overall message was: "Titan is awesome"

* This was our first foray into the world of graphs; we are not experts by any stretch of the imagination

* These slides were meant as talking points, so they may not come across as intended when taken out of context

* I've been informed that Titan can handle supernodes just fine: http://thinkaurelius.com/2012/10/25/a-solution-to-the-supernode-problem/

Cheers,
--Abhi

Transcript

1. 1 Titan: The Journey Oct 28, 2013 Abhinav  Ajgaonkar   @abh1nv   abhi@crowdriff.com Kent  English   @kentenglish   kent@crowdriff.com
2. Why Titan?
3. 3 But first of all, why a graph?
4. 4 Social networks are a canonical example of data that is well suited to graph representation
5. 5 CrowdRiff is interested in the relationships established between people through activities
6. 6 So a graph is a natural fit but then this happens…
7. 7 So why a distributed graph database?
8. 8 Why Titan specifically? ! • Active,  responsive  and  competent  team   • Choice  of  backend  storage  (more  on  this  later)   • Faunus   • Fulgora  (soon)
9. 9 The Aurelius Team Dr.  Marko  A.  Rodriguez     • TinkerPop  cofounder  and  lead  developer  of  the  Gremlin  graph   traversal  language  and  Faunus  graph  analytics  engine   ! Dr.  Matthias  Broecheler   • award-­‐winning  research  includes  high  performance  index  structures   and  query  answering  algorithms  for  graph  structured  data   ! From  http://thinkaurelius.com/team/
10. 10 What is Faunus? • Scalable,  distributed  global  graph   processing   • Analyzes  graphs  using  a   MapReduce  implementation  of  the   Gremlin  graph  traversal  language
11. 11 What is Fulgora (going to be)? In-­‐memory  data   processing  for  low  latency   query  answering  of  both   OLTP  (real-­‐time)  and  OLAP   (batch)  queries.   ! (Think  Apache  Giraph)
12. Titan is not a Graph Database
13. A Production Deployment Consists of: 1. Storage Backend 2. Titan Library 3. A server that ties it all together
14. Pluggable Storage Backends Image  credit:  https://github.com/thinkaurelius/titan/wiki/Storage-­‐Backend-­‐Overview
15. Guess which one we picked?
16. New in 0.4.0 Image  credits:     http://www.hazelcast.com/images/logo.png   https://github.com/thinkaurelius/titan/wiki/Using-­‐Persistit
17. A Production Deployment Consists of: 1. Storage Backend 2. Titan Library 3. A server that ties it all together
18. Titan Server
19. Embedded Mode Rexster + Titan + Cassandra running in a single JVM Image  credit:  https://github.com/thinkaurelius/titan/wiki/Using-­‐Cassandra
20. Remote Cassandra Mode Rexster + Titan with a remote Cassandra cluster Image  credit:  https://github.com/thinkaurelius/titan/wiki/Using-­‐Cassandra
21. The CrowdRiff Setup Custom Query Engine with a remote Cassandra cluster (using the Titan Java Driver)
22. Gotchas
23. Titan releases are backwards incompatible :( (for  now)
24. Edge Retrievals are not O(1) (by  design) Translation:  Don’t  even  think  about  creating  supernodes.
25. Indexing is tricky (choose  your  indexes  very  carefully) For  example:     A  secondary  index  (via  Elastic  Search  or  Lucene)  is  required  for  enabling  range  queries.   ! Elastic  Search  requires  that  the  field  must  be  unique  in  some  direction  and  be  present  in  all   vertices.  This  would  not  be  useful  for  indexing  a  timestamp  field.
26. Property Types are forever (even  those  you  don’t  declare  explicitly)
27. Locking Exceptions in a highly concurrent environment (i.e.  don’t  forget  your  retry  logic)
28. Faunus dislikes Supernodes
29. Questions?