Challenges when scaling: continued adventures in Swift's sharding

Talk on how the sharding of the container metadata layer is progressing which will allow you to store huge amounts of objects in a single Openstack Swift container.

Transcript

1. Challenges when scaling: Continued Adventures in Swif’s Sharding Matthew Oliver <matt@oliver.net.au>
2. 2 • Sofware Developer at Rackspace Australia • Openstack Swif Core • Been attending LCA since Melbourne 2008 Who am I
3. 3 • Swif overview • Problem description • Journey so far – POC 1 - 3 • Reaching the technical limit on some technology choices – POC 4 • Eventual consistency strikes – POC 5 Overview
4. 4 In essence there are 3 resources in Swif, the account, container and object. Swif overview - Accounts, Containers and Objects GET /v1/AUTH_matt/LCA2017/example.png
5. 5 Swif overview - Topology 101
6. 6 Swif overview - Topology 101
7. 7 Swif overview - Topology 101
8. 8 Swif overview - Topology 101
9. 9 Problem Description
10. 10 Journey so far.. Or recap from ACT 1 Proof of concepts: 1. Object hashing shard_no = md5(object) >> shard_power
11. 11 Journey so far.. Or recap from ACT 1 Proof of concepts: 1. Object hashing 2. Distributed prefix trees
12. 12 Journey so far.. Or recap from ACT 1 Proof of concepts: 1. Object hashing 2. Distributed prefix trees 3. Pivot Points/Split tree
13. 13 Finding the pivot point was very simple.. But doesn’t scale: POC #3 - What went wrong SELECT name FROM object WHERE deleted=0 LIMIT 1 OFFSET ( SELECT object_count / 2 FROM policy_stat);
14. 14 POC #3 - What’s wrong with OFFSET? In SQLite OFFSET under the hood really is just: LIMIT x OFFSET y LIMIT x + y (and discard the first y values).
15. 15 POC #3 - What’s wrong with OFFSET? In SQLite OFFSET under the hood really is just: LIMIT 1 OFFSET 350 000 000 LIMIT 1 + 350 000 000 (and discard the first 350 million values).
16. 16 POC #4 - Snipping
17. 17 POC #4 - Snipping
18. 18 POC #4 - Snipping
19. 19 POC #4 - Snipping
20. 20 POC #4 - Snipping
21. 21 POC #4 - Eventual consistency strikes again! At any point in time, other primary nodes could be in any other state. Any action we do on a container database needs a timestamp so at some point all primary nodes coalesce into the correct final state. Deleting rows, afer moving to a shard, means writing rows to the large database.
22. 22 POC #5 - Read only db Container DB Container DB Pivot DB New database can store: • Metadata • References to pivots • Holding table for objects
23. 23 POC #5 - Read only db Container DB Container DB Pivot DB Pivot DB UNSHARDED SHARDING SHARDED
24. 24 POC #5 - Update requests
25. 25 POC #5 - Update requests
26. 26 POC #5 - Update requests
27. 27 POC #5 - Update requests
28. 28 POC #5 - Update requests
29. 29 POC #5 - Container GET - Sharded
30. 30 POC #5 - Container GET - Sharding
31. 31 POC #2 - Distributed Prefix Trees (Tries)
32. 32 POC #2 - Distributed Prefix Trees (Tries)
33. Image FPO 33 Questions? IRC: mattoliverau Channel: #openstack-swift on Freenode Email: matt@oliver.net.au or matthew.oliver@rackspace.com Twitter: @mattoliverau