Torus: Storage Demystified

My first talk on the inspiration and implementation of Torus

Transcript

1. Barak Michener @barakmich | barak.michener@coreos.com | github. com/coreos/torus Torus: Storage Demystified
2. Let’s talk about I/O conceptually
3. UNIX File API • read(), write() • fsync() (Flush(), in Go) The fsync() function is intended to force a physical write of data from the buffer cache, and to assure that after a system crash or other failure that all data up to the time of the fsync() call is recorded on the disk. Since the concepts of ``buffer cache'', ``system crash'', ``physical write'', and ``non-volatile storage'' are not defined here, the wording has to be more abstract.
5. Caches all the way down • App • Filesystem • Block Device • Hardware
6. Your Choice Speed Durability
7. What’s the dream?
8. Magical File Pointer • Distributed/Sharded • Replicated • Error-corrected • Versioned • Available anywhere on the cluster The Dream of Plan 9 is alive in Golang
9. Storage is Generic (when you squint) Bytes, persisted somewhere • Block Devices ◦ One, fixed size, “magical file pointer” • Object Stores ◦ HTTP API to some magic bucket of (often append- only) MFPs • Filesystems ◦ POSIX semantics and metadata referring to MFPs
10. Reminder : The only way to know something is durable is to sync() (and pray you’re not being lied to)
11. Why rebuild?
12. Storage is a HARD problem. ...But Torus’ major claim is that it’s a SEPARABLE problem.
13. We all agree on the architecture Metadata Storage Storage Storage Storage Client
14. Architecture, Explained • Client ◦ Mounts a data source ◦ Gets data from storage directly ◦ Updates metadata • Metadata ◦ Presents a consistent view for multiple clients ◦ Knows where the data lives ◦ Service discovery, etc • Storage ◦ Truthfully stores data it’s responsible for
15. Architecture, Explained • Client <-> Metadata ◦ Updates metadata about what it’s reading/writing ◦ Where to find the storage in question • Client <-> Storage ◦ Bulk data transport • Metadata <-> Storage ◦ Is told what data it owns ◦ Tells it what to keep or throw away
16. Distributed Consensus is Hard Any form of distributed storage (FS, database, ….) requires at least some of its data be strongly consistent. Everyone has had to reimplement this in some form or another, if for no other reason than locking -- but for many good reasons too, eg, declaring something written/available to all clients. Yo, turns out we have a secret weapon….
17. So what’s the separation? Storage Storage Storage Storage Client
18. “That means what is left is the business logic of storing bits in ways which are accessible, redundant, scalable, and performant. While this is no small feat, being able to assume and trust the consistency of your metadata makes the storage logic much easier” --Seán McCord
19. Demo Time!
20. Let’s look at the implementation
21. • Discovery of Peers • Coordination ◦ Checkpointing ◦ Metadata • Takes the place of MDS Server • Entry point for main() ◦ Create a Server object ◦ Creates Files • Monitoring • Heartbeats Distributor P2P GRPC Storage INode Abstraction • Manages Peer-to-Peer GRPC • Rebalancing, GC Thread • Stores the actual data blocks Torus process etcd
22. • Discovery of Peers • Coordination, etc Server Distributor P2P GRPC Storage • No long-term storage, all data comes from the P2P Torusblk process etcd Block Volume Abstraction • Treats a Server File as a block device • Provides interface with other services (NBD) • Locks the volume • Same code for the Server and Distributor
23. INode BlockRef BlockRef BlockRef CRC Reed-Solomon Encryption Replaces INode ID Inside a File
24. INode v1 INode v2 sync() BlockRef BlockRef BlockRef BlockRef BlockRef BlockRef Data Data Data Data Ready for GC at versions >=2
25. Pure func() [Node2, Node4, Node1, Node3] BlockRef This is a full permutation of the nodes. They may not all have the data, but the first N (replication leve) will eventually hold the data when quiescent. F(List of Nodes, Replication factor)
26. Pure funcA() [Node2, Node4, Node1, Node3] BlockRef Pure funcB() [Node2, Node5, Node4, Node1, Node3] BlockRef
27. [Node2, Node5, Node4, Node1, Node3] R=2 [Node2, Node4, Node1, Node3] R=2 [Node2, Node5, Node4, Node1, Node3] R=2 [Node2, Node5, Node4, Node1, Node3] R=2 [Node2, Node5, Node4, Node1, Node3] R=2
28. Where do we go from here?
29. Short term • Stress testing • More block device features • Even better Kubernetes integration
30. Long term • Object storage volumes ◦ Multi R/W, various object-y APIs • Experiments with transports ◦ eg, QUIC • More intricate rings ◦ Rack-awareness, data locality, drive speed…. ◦ (Lots of ‘fun’ here) • Kubernetes awareness ◦ Schedule work near data
31. Thank you! Barak Michener @barakmich | barak.michener@coreos.com | github. com/coreos/torus We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers Including Torus!
34. coreos.com/fest - @coreosfest May 9 & 10, 2016 - Berlin, Germany
35. We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers 90+ Projects on GitHub, 1,000+ Contributors OPEN SOURCE CoreOS.com - @coreoslinux - github/coreos Secure solutions, support plans, training + more ENTERPRISE sales@coreos.com - tectonic.com - quay.io CoreOS is Running the World’s Containers
36. $ rkt run presentation # everyone claps # drink a beer $ rkt run...
37. $ rkt run presentation # everyone claps # drink a beer $ rkt run... Let’s run a container Man that was fun Supports pods for fun in a group setting
38. Content Title Subpoints, more content Subpoints, more content More content and more context, too.
39. Some logo files you may want to use:
40. Content Title Subpoints, more content Subpoints, more content More content and more context, too.
41. Some logo files you may want to use: