How we stored JSON in Cassan

Transcript

1. Store JSON in Cassandra the Hard Way Josh Dzielak @dzello A Personal Data Modeling Journey 07/23/2014 #CassandraSF
2. I’m Josh
3. I’m Josh
4. I’m Josh Formerly
5. I’m Josh Formerly
6. I’m Josh Formerly
7. I’m Josh Formerly
8. Our Hosts
9. This talk is about data modeling row key column name column name . . . column value column value . . .
10. This talk is also about
11. A personal
12. intense
13. rewarding
14. Journey
15. designing a new, complex distributed system
16. out of numerous other quasi-documented heterogenous distributed systems.
18. Whee
19. The Hype Cycle of New Technologies
20. Hype Cycle Example - WWW
21. Hype Cycle Example - WWW Peak: Democratized information for All… World Peace!
22. Hype Cycle Example - WWW Trough: Slow Connectivity / Walled Gardens
23. Hype Cycle Example - WWW Slope: Broadband / Web 1.0, 
 Web 2.0 + Social
24. Hype Cycle Example - WWW Today: The Web is Stable, Productive Platform
25. The Hype Cycle Applies to Projects Too
26. Peak of Inflated Expectations
27. Peak of Inflated Expectations Shiny Egg!
28. Peak of Inflated Expectations • Faberge egg is gorgeous
29. Peak of Inflated Expectations • Faberge egg is gorgeous • Prototype is gorgeous
30. Trough of Disillusionment
31. • Faberge egg is fragile Trough of Disillusionment
32. • Faberge egg is fragile • Prototype is fragile Trough of Disillusionment
33. Trough of Disillusionment
34. Trough of Disillusionment Rabbit Hole < …
35. Trough of Disillusionment Sink Hole < …
36. Trough of Disillusionment Zombie Hole!
37. Whee
38. Deep Breaths
39. Drawing Board
40. Slope of Enlightenment
41. Slope of Enlightenment Starts with…
42. Slope of Enlightenment An moment!
43. Productivity
44. Productivity Stable predictable progress
45. Hype Cycle In Photos
46. Hype Cycle In Photos
47. Hype Cycle In Photos
48. Hype Cycle In Photos
49. Hype Cycle In Photos
50. Hype Cycle in Code
51. Hype Cycle in Code Peak
52. Hype Cycle in Code Peak Trough
53. Hype Cycle in Code Peak Trough Slope
54. Hype Cycle in Code Peak Trough Slope Productivity!
55. Hype Cycle Commit Messages
56. Hype Cycle Commit Messages
57. Hype Cycle Commit Messages
58. Hype Cycle Commit Messages
59. Hype Cycle Commit Messages
60. Hype Cycle Commit Messages
61. Hype Cycle Commit Messages Sorry
62. The Hype Cycle applies to building distributed systems
63. The Hype Cycle applies to building distributed systems Like the one at Keen IO
64. Keen IO
65. Keen IO Analytics API for Developers
66. Keen IO Analytics API for Developers <3
67. 2012 - 2013 •Speed to market •Great for JSON •Strong Community Keen IO
68. •Linear Scale •Operational Story •Strong Community 2013 … ? Keen IO
69. +
70. +
71. But wait
73. JSON?
74. wat JSON?
75. Cassandra does not have a JSON data type
76. Cassandra does not have a JSON data type What do we do?
77. { "id": “c645-abd3“, "timestamp": "2013-08-06T12:30:00-0700Z", "url": "https://keen.io", "method": "GET", "response": { "status": 200 } } Example JSON Event Models an HTTP request
78. Use static column families?
79. Use static column families? CREATE TABLE requests ( KEY uuid PRIMARY KEY, timestamp text, url text, response_status integer, … )
80. Use static column families? • Possible if schema is known in advance CREATE TABLE requests ( KEY uuid PRIMARY KEY, timestamp text, url text, response_status integer, … )
81. Use static column families? • Possible if schema is known in advance • Still might not be fast to scan +1M / +1B rows CREATE TABLE requests ( KEY uuid PRIMARY KEY, timestamp text, url text, response_status integer, … )
82. Use static column families? • Possible if schema is known in advance • Still might not be fast to scan +1M / +1B rows • Not possible with Keen - schema is implicitly defined by customers, and not enforced at write-time CREATE TABLE requests ( KEY uuid PRIMARY KEY, timestamp text, url text, response_status integer, … )
83. Use static column families? • Possible if schema is known in advance • Still might not be fast to scan +1M / +1B rows • Not possible with Keen - schema is implicitly defined by customers, and not enforced at write-time • Keen has 500,000+ user-defined collections today, stored in 1 Cassandra column family CREATE TABLE requests ( KEY uuid PRIMARY KEY, timestamp text, url text, response_status integer, … )
84. …
85. 3 Perfectly Plausible Attempts to Store JSON in Cassandra
86. 3 Perfectly Plausible Attempts to Store JSON in Cassandra And The 1 That Worked
87. requests cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 400 } . . . Row Key UTF8Type User’s collection name (w/ Project ID prefix, omitted for brevity) ! Column Name Composite(UTF8Type, TimeUUID) A unique UUID for the event, plus a TimeUUID CF: events
88. Good Simple write path, can filter by timeframe CF: events requests cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 400 } . . .
89. Good Simple write path, can filter by timeframe •Row growth is unbounded; hot spots for big collections •Can only filter on time by > or <, not both •Little gains from parallelizing b/c row is the same •JSON deserialization in queries is expensive Bad CF: events requests cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 400 } . . .
90. requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . . CF: events
91. • Turned ‘requests’ into ‘requests-[0…n]’ requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . . CF: events
92. • Turned ‘requests’ into ‘requests-[0…n]’ • No more unbounded row growth requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . . CF: events
93. • Turned ‘requests’ into ‘requests-[0…n]’ • No more unbounded row growth •Obvious, effective way to parallelize queries requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . . CF: events
94. • Turned ‘requests’ into ‘requests-[0…n]’ • No more unbounded row growth •Obvious, effective way to parallelize queries •Required another CF, an ‘index’ to store pointers to these ‘buckets’: requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . . requests 0 1 . . . { “start” : “2014-07-21” … } { “start” : “2014-07-22” … } . . . CF: index CF: events
95. Concerns CF: events requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . .
96. •Write path has become more complex; position of bucket index must be kept in another CF and rolled over Concerns CF: events requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . .
97. •Write path has become more complex; position of bucket index must be kept in another CF and rolled over •Keeping consistent state for more than 1 CF 
 (solution: atomic batch) Concerns CF: events requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . .
98. •Write path has become more complex; position of bucket index must be kept in another CF and rolled over •Keeping consistent state for more than 1 CF 
 (solution: atomic batch) •Still have to deserialize JSON at query time Concerns CF: events requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . .
99. •Write path has become more complex; position of bucket index must be kept in another CF and rolled over •Keeping consistent state for more than 1 CF 
 (solution: atomic batch) •Still have to deserialize JSON at query time •Still not fast enough Concerns CF: events requests-0 cef7-be80:TimeUUID() a87b-472c:TimeUUID() . . . { “status” : 200 } { “status” : 304 } . . . requests-1 9f45-bf97:TimeUUID() 76ab-dca6:TimeUUID() . . . { “status” : 417 } { “status” : 200 } . . . requests-n . . . . . . . . .
100. We are here
101. We are here Why?
102. Our assumptions about Cassandra were not valid.
103. Our assumptions about Cassandra were not valid. Namely, that it would do most of the work for us.
105. - Dieter Rams
106. The Epiphany
107. The Epiphany Don’t let the physical data model dictate the logical data model.
108. The Epiphany Don’t let the physical data model dictate the logical data model. Define an ideal logical model first. Then find a way to implement it using a physical data store.
109. The Epiphany Don’t let the physical data model dictate the logical data model. Define an ideal logical model first. Then find a way to implement it using a physical data store. This was not obvious coming from the relational or document store worlds.
110. The Epiphany Don’t let the physical data model dictate the logical data model. Define an ideal logical model first. Then find a way to implement it using a physical data store. This was not obvious coming from the relational or document store worlds. It felt dirty. But it doesn’t have to be.
111. Don’t store JSON as JSON! The Epiphany
112. requests-0 timestamp response.status url [“2014-07-21…”, “2014-07-21…”] [200, 400] [“keen.io”, “keen.io”] requests-1 timestamp response.status url [“2014-07-21…”, “2014-07-21…”] [200, 400] [“keen.io”, “keen.io”] Row Key UTF8Type User’s collection name with a ‘bucket’ sequence number ! Column Name UTF8Type The dotted.name of the property ! Column Value BytesType A Kryo-serialized, compressed Object[] containing ~5000 property values CF: events
113. requests-0 timestamp response.status url [“2014-07-21…”, “2014-07-21…”] [200, 400] [“keen.io”, “keen.io”] requests-1 timestamp response.status url [“2014-07-21…”, “2014-07-21…”] [200, 400] [“keen.io”, “keen.io”] Notes Order of properties in the Object[] arrays is *very* important! ! timestamp[5], response.status[5] and url[5] must be properties from the same event CF: events
114. Super-fast queries due to: • columnar access • columnar compression • Kryo deserialization (fast!) • parallelism via bucketing The Good
115. Super-fast queries due to: • columnar access • columnar compression • Kryo deserialization (fast!) • parallelism via bucketing Tradeoffs • Application code is more complicated. • Writes and reads must understand this structure. The Good
116. Super-fast queries due to: • columnar access • columnar compression • Kryo deserialization (fast!) • parallelism via bucketing Tradeoffs • Application code is more complicated. • Writes and reads must understand this structure. The Good Worth it!
117. Whee
118. Lessons Learned • Set performance targets early on ! • Design a logical data model first with the characteristics you need (columnar, compressible, partition-able), then figure out how to project it onto physical storage. ! • Don’t be afraid to try crazy stuff that would feel unnatural in the relational or document worlds
119. 3 Awesome Things That Happened In Production That Were Not Awesome At The Time
120. Doomstones A machine with corrupted RAM propagated future dated tombstones around the ring. This made data ‘invisible’ for number of row keys.
121. Serialization Incompatibility Code change that changed serialization format made it into write path before read path. 
 Result: KryoException
122. Clock Drift When ring clocks are not in sync “Last Write Wins” becomes “Any Write Wins”.
123. Thanks! ! Questions? Talk at my face ! or email me at josh@keen.io