Object storage with Gluster

Object storage with Gluster using using OpenStack Swift

Transcript

1. Object storage with Gluster Using OpenStack Swift Prashanth Pai ppai@redhat.com
2. Openstack Swift a highly available, distributed, eventually consistent object store. • Alternative to AWS S3 • Decoupled from OpenStack • Suitable for unstructured data • Scales horizontally • But you can't mount it! Logical Hierarchy Accounts/Containers/Objects
3. gluster-swift ➔ Just another application using fuse mount ➔ Uses and “plugs into” OpenStack Swift ➔ Against having to write our own object front-end ➔ “Turns off” Swift’s distribution and replication
4. Object access trivia (swift) ➔ Object = {Data + Metadata + Identifier} ➔ Abstracted ➔ Stateless (HTTP) ➔ No partial PUTs ➔ Ranged GETs supported ➔ No renames, only COPY ➔ Fixed logical hierarchy
5. Vanilla Swift Ring SQLite DB SQLite DB XFS Account Server Container Server Object Server Proxy Server Client HTTP
6. Pluggability in Swift • Backends implement DiskFile • Object server is pluggable officially. • Account and container code are not. • Makes cross-version compatibility tricky.
7. Gluster-Swift Ring Account Server Container Server Object Server Proxy Server Client Same node FUSE mount
8. Proxy Server - pipeline /etc/swift/proxy-server.conf pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server ➔ Very similar to translators in gluster ➔ Middlewares are highly modular ➔ Many features are implemented as middlewares
9. Amazon S3 API Support ➔ Implemented as a middleware (swift3) ➔ Converts incoming S3 requests into Swift requests S3 Request Swift Response Swift3 middleware Swift Request S3 Response Proxy Server S3 Client
10. Swift Object Ops → Fops PUT getxattr() mkdir()* open() write() fsetxattr() fsync() rename() close() GET open() fstat() fgetxattr() read() close() POST getxattr() stat() setxattr() DELETE getxattr() stat() unlink() rmdir()* *multiple calls - recursive HEAD getxattr() stat()
11. # getfattr -d /mnt/gluster-object/test/c1/animals/cat/kitten user.swift.metadata="{ \"Content-Length\":\"4\", \"Etag\":\"4a4be40c96ac6314e91d93f38043a634\", \"X-Timestamp\":\"1475214280.33138\", \"X-Object-Type\":\"file\", \"X-Type\":\"Object\", \"Content-Type\":\"application/x-www-form-urlencoded\"}" # curl -i http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT -d'meow' HTTP/1.1 201 Created Last-Modified: Fri, 30 Sep 2016 05:44:41 GMT Content-Length: 0 Etag: 4a4be40c96ac6314e91d93f38043a634 Content-Type: text/html; charset=UTF-8 X-Trans-Id: txae858eb9169a4e1eb2a09-0057edfbc8 Date: Fri, 30 Sep 2016 05:44:40 GMT Object PUT
12. • Atomicity • open() with O_TMPFILE + linkat() • mkstemp() and rename() That DHT trick that every user should know about: .filename.suffix hashed to same brick as filename .kitten.jpg.4e91d93f380 kitten.jpg → • Multiple client PUTs ? • Last write wins Object PUT
13. Swift Listing Ops → Fops Container GET (plain text) – only names getxattr() opendir() readdir()* getxattr()1 Container GET (json/xml) – names + metadata getxattr() opendir() readdir()* getxattr()2 *multiple calls – recursive 1 only on dirs 2 on every inode
14. Swift API Deviations # curl -i http://localhost:8080/v1/AUTH_test/c2 -X GET HTTP/1.1 200 OK Content-Length: 7 X-Container-Object-Count: 2 Accept-Ranges: bytes X-Storage-Policy: Policy-0 X-Container-Bytes-Used: 0 X-Timestamp: 0000000001.00000 Content-Type: text/plain; charset=utf-8 X-Trans-Id: tx18e21ccd03f04f4698a1a-0057ed1b84 Date: Thu, 29 Sep 2016 13:47:48 GMT HTTP Header needs to go first, then the response body.
15. Swift API Deviations # curl http://host:8080/v1/AUTH_test/c1/animals/cat -X PUT Now, cat is a file # curl http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT Here, cat is supposed to be directory but it’s already a file. Backend object path on filesystem gluster-swift: /mnt/gluster-object/test/c1/animals/cat Vanilla Swift: /mnt/sdb1/objects/778/69f/c2b307d78b6c419c0c1b76d91c08c69f/1412628708.01757.data
16. ➔ Zero copy and copy offload ➔ Use libgfapi python bindings ➔ Use hummingbird project ➔ Native object ? glfs_put(..), glfs_get(..) Ideas
17. Demo ? https://asciinema.org/a/88026