Data Structures: The Code That Isn’t There

Watch the presentation at: http://www.infoq.com/presentations/Data-Structures

Most programmers rely on a few core data structures, but they’re missing out on useful properties that more specialized data structures provide.

The wrong data structures can bog implementation down in irrelevant detail or create behaviors which waste time and effort, but the right ones can give powerful guarantees for free. My talk will present lesser-known data structures and their unique advantages:

- Skiplists are simple data structures whose design leads to balanced binary tree-like performance, without any need for non-localized operations such as rebalancing. (Example use case: Demonstrating how simple invariants can lead to powerful emergent properties.)

- Difference lists provide a way to explicitly model temporary uncertainty. They are immutable, yet can still be refined as more information becomes available. They have much in common with lazy evaluation, but for data rather than control flow. (Example use case: Adding more flexibility to immutable languages by relaxing the flow of time.)

- Rolling hashes can find deterministic breaking points in buffers of binary data, enabling consistent chunking and re-use as data changes. (Example use case: rsync.)

- Jumpropes (a data structure of my own invention) automatically de-duplicate content stored in them, including data shared between multiple files. Modified content can be stored with very little additional overhead, allowing for cheap versioning. Finally, the next several fragments can always be retrieved in parallel, enabling simple buffering for streaming media. (Example use case: scatterbrain, a distributed filesystem to be released soon.)

Transcript

1. the code that isn’t there Data Structures
2. “A data structure is just a stupid programming language.” - Ralph William Gosper Jr.
3. “A data structure is just a stupid programming language tiny virtual machine.”
4. Fundamentals: linked list array hash table binary tree data structures
5. using list for a set? ruby’s require, circa 1.9.2 bit.ly/krkvJP data structures
6. FAIL
7. "The cheapest, fastest, and most reliable components are those that aren't there." data structures - Gordon Bell
8. ...set the path of least resistance. data structures
9. implementation details bubble up to the surface data structures
10. git vs. hg data structures
11. internal data formats data structures
12. data structures
13. but, enough about the basics... jarring segue
14. skiplists
15. first, take an ordered linked list... skiplists
16. add an “express lane”... skiplists
17. add another... skiplists
18. skiplists skiplists
19. skiplists
20. but, how do we balance that? skiplists
21. real trees are not balanced, skiplists just balanced enough
22. random levels probability distribution skiplists
23. skiplists
24. skiplists
25. controlled trade-offs skiplists
26. “roundabouts, not traffic lights” philosophy
27. traffic light philosophy
28. roundabout philosophy
29. global order emerges from local decisions philosophy
30. only immediate neighbors are affected most locking is local, little contention philosophy
31. still: mutation? skipping on
32. still: mutation? isn’t Rich Hickey in the building? skipping on
33. difference lists
34. ?- uses(prolog, Person). no difference lists
35. retroactive immutability! difference lists
36. Unification ?- [X, Y, X] = [1, 2, Z]. X = 1 Y = 2 Z = 1 yes difference lists
37. data structures with “holes” in them difference lists
38. logic variables: explicitly modeling partial information difference lists
39. Unification ?- [X, Y, H] = [1, 2, Z]. X = 1 Y = 2 Z = H ??? yes difference lists
40. constant-time append to an immutable list difference lists
41. difference lists
42. ?- A = [1,2|B], B = [3|C] A = [1, 2, 3|C] yes difference lists
43. ?- A = [1,2|B], B = [3|C] A = [1, 2, 3|C] yes difference lists Yes!
44. still immutable ...just unstuck in time difference lists
45. more fundamental than lazy evaluation or lazy streams (a bit closer to futures/promises) difference lists
46. ...apply to more than just lists difference trees? difference dictionaries? difference lists
47. rolling along...
48. find matching/overlapping sequences in binary data rolling hash
49. hash everything against everything? md5, sha1, etc. are too slow rolling hash
50. “Fast, Cheap, and Able to Roll” rolling hash
51. The_quick_brow 0xba5eba11 he_quick_brown 0x0b5e55ed e_quick_brown_ 0xdeadbeef _quick_brown_f 0xf005ba11 quick_brown_fo 0xbea2ded1 uick_brown_fox 0xdefea7ed rolling hash
52. fill then window buffer, then: drop 1, take 1, new hash repeat rolling hash
53. rsync rolling hash
54. sync data across slow network minimize passes & bandwidth rolling hash
55. just send the hash for each block, eh? S D rolling hash
56. not so fast: inserts/deletes shift each block! S D rolling hash
57. Match with a rolling hash, then sha1 S D rolling hash
58. request unmatched regions S D rolling hash
59. (for more details, check out Andrew Tridgell’s thesis: Efficient Algorithms for Sorting and Synchronization, pg. 64) rolling hash
60. can also be used for chunking data rolling hash
61. even with changes, falls back in sync X rolling hash
62. ...finds deterministic breaks ...cheaply matches blocks rolling hash
63. jumprope
64. storage for large binary strings (where “string” can mean “file”) jumprope
65. “content addressable storage” no pointers, easy distribution jumprope
66. persistent and immutable can be freely cached jumprope
67. de-duplication for free jumprope
68. ...kind of like a git repo jumprope
69. three structural elements: jumprope
70. leaf jumprope
71. limb jumprope
72. trunk jumprope
73. then, it just needs a key/value store get(hash) => jr_node set(hash, jr_node) => OK | ERROR jumprope
74. building it up jumprope
75. add another jumprope
76. and another jumprope
77. grow a level, then add a limb jumprope
78. jumprope
79. seek into it jumprope
80. duplicated nodes are shared free de-duplication jumprope
81. low metadata overhead jumprope
82. fetch it (in parallel) stream it mirror it jumprope
83. use case? scatterbrain jumprope
84. randomization persistence & immutability emergence from local behavior lack of pointers (CAS) properties
85. We’re Hiring!
86. the php hammer: http://www.flickr.com/photos/raindrift/sets/ 72157629492908038/ jumprope: http://www.flickr.com/photos/adwriter/5826718940/ traffic light: http://www.flickr.com/photos/bexross/2636921208/ cc attribution
87. roundabout: http://www.flickr.com/photos/trendscout/4000299787/ tree: http://www.flickr.com/photos/neilspicys/2348959159/ leek: http://www.flickr.com/photos/richardnorth/7319091956/ cc attribution
88. Scott Vokes @silentbicycle github.com/silentbicycle Questions?