Zen and the Art of Storage Services

Transcript

2. Raghavendra Prabhu (RVP) Zen: Pinterest’s Graph Storage Service
3. “Given how robust the messenger is on day one, it’s surprising to learn that Pinterest built the entire product 
 in three months.” — The Verge
5. What does it take to do this consistently? Many different things •Hire the best •Culture •Focus Infrastructure that doesn’t get in the way
6. Challenge for Infrastructure
7. Our approach •Make it part of the team mission statement •Design systems with ‘move fast’ in mind •Separation of concerns: feature vs reliability
8. Persistent Storage Even with a distributed database, app needs to deal with: •Schema design •Fault tolerance •Capacity management •Performance tuning
9. Solution 1: UserMetaStore Storage-as-a-Service: Key-value thrift API on top of HBase Features: •Key partitioning to balance load •Master-slave clusters, semi automatic failover •Speculative execution •Multi-tenancy with traffic isolation
10. Storage-as-a-service is a great step forward, but can we do better?
11. Example: Messages Data Model Conversation Message 1 Message 2 Message N User User Participates Contains
12. Realization •These object models closely resemble a graph •Objects are nodes, edges represent relationships •Typical needs: • retrieve data for a node or edge • get all outgoing edges from a node • get all incoming edges from a node • count incoming or outgoing edges for a node
13. Enter Zen! •Provides a graph data model instead of key-value •Automatically creates necessary indexes •Materializes counts for efficient querying •Implemented on top of HBase, but can plug in other backends
14. What Zen is NOT •NOT a full fledged graph database •NO advanced graph operations •Basically an object-relationship data model on top of existing databases to simplify app development
15. Zen API Nodes: • addNode, removeNode, getNode • Node id: globally unique 64-bit integer
 ID 123 Prop 1 Val 1 Prop 2 Val 2
16. Zen API Edges: • addEdge, removeEdge, getEdge • Edge Ref: (edgeType, fromId, toId) • Score for ordering Edge Ref 120, 123, 4567 Prop 1 Val 1 Prop 2 Val 2
17. Zen API Edge Queries: • getEdges, countEdges, removeEdges struct EdgeQuery {! ! 1: required NodeId nodeId;! ! 2: required EdgeDirection direction;! ! 3: optional TypeId edgeType;! ! }
18. Zen API Property Indexes •Unique index •Ensures a property value is unique across all nodes of a type •Non-unique index •Allows retrieval by property value •Works for both nodes and edges
19. Zen API Type System •Declare node and edge types •Specify type schema, e.g. unique and non-unique index properties •Fully online: no deploy, no config •Internally implemented on top of Zen itself!
20. Illustration: Messages on Zen Id:1234 Id:2345 Id:3456 Type: Participates Type: Contains Type: Conversation Started: 12 Aug 2014 08:00 Header: “Great pin!” Pin Id: 10001 [non-unique] Type: User Name: “Ben Smith” [unique] Status: Active Type: Message Sent: 12 Aug 2014 08:00 Text: “Great pin!”
21. Zen: Current Usage Products: • smart feed, messages, network news, interest graph and other upcoming features ! Numbers: • ~10 clusters • 100,000+ requests per second at peak • Over 5 million HBase operations per second
22. Xun Liu Internals and Production Learnings
23. Zen Backends •HBase backend implemented in fall 2013 •Currently working on MySQL backend •Other potential backends in future
24. HBase Data Model Overview Data
25. HBase Data Model Overview Data col1 col2 row-key-1 val1 val2
26. HBase Data Model Overview Data col1 col2 col3 row-key-1 val1 val2 row-key-2 val3 val4
27. HBase Data Model Overview Data col1 col2 col3 col4 row-key-1 val1 val2 row-key-2 val3 val4 row-key-3 val5
28. Zen - Property Data type name score distance 12345 (node) 10 Ben Smith 12345-20-67890 (edge) 1000 1 mile
29. Zen - Property Index Data ID <hash>-unique-10-name=ben smith 12345 <hash>-nonuniq-10-lastname=smith-12345 <hash>-nonuniq-10-lastname=smith-67890
30. Zen - Edge Score Index Data 12345-out-20-1000-67890 12345-out-20-1001-67891 12345-in-30-990-67892 12345-in-30-991-67893
31. Zen - Edge Count Data Count 12345-out-20 2 12345-in-30 4
32. Status - Soft Delete New Features
33. Built-in Cache New Features Zen Cache HBase Client Zen HBase Client Cache Before After
34. Namespace New Features Node Namespace 1 Edge Index Node Namespace 2 Edge Index
35. New Features •Online type schema change •Optional reverse edge •Optional edge count •Retrieval of subset of properties •Descending edge score
36. Performance Work Demanding work load needs special tuning • Inserting 1 million edges per second • Excessive HLog (WAL) flushes
37. Performance Work Batching • Client Side Batching — bulk edge insertion • Zen Server Side Batching — buffer edits across clients & flush together • Reduced HLog (WAL) flushes by orders of magnitude
38. Performance Work Memory v.s. Performance • Bloom filter • reduce disk seeks • memory cost: 1 byte per row • Block size • the smaller block size the better random access performance • memory cost: bigger index size
39. Performance Work CPU v.s. Data Size • Encoding • FAST_DIFF: effective in reducing data size, cpu intensive • PREFIX: less effective in size reduction, less cpu intensive • Compression • SNAPPY, LZO, GZ, etc
40. Performance Work Capability to tune storage engine per special load Zen production setup • Dedicated Zen cluster • Namespace in shared Zen cluster
41. Data Consistency Add an edge 1. CAS create the edge row and properties 2. CAS create the unique index if any 3. Create non-unique index if any 4. Create edge score index for outgoing direction 5. Create edge score index for incoming direction 6. Increment edge count for outgoing direction 7. Increment edge count for incoming direction
42. Distributed transaction or not?
43. Data Consistency Stay on top of data inconsistencies • Manual rollback in Zen server • Offline jobs (Dr Zen) to scan and fix inconsistencies • Tools to debug and fix one-off inconsistency
44. Future Work •Dr Zen (make it more efficient) •Other backends: MySQL, etc •Distributed transactions •Open source!