Object storage with Gluster Using OpenStack Swift Prashanth Pai [email protected] 

Openstack Swift a highly available, distributed, eventually consistent object store. ● Alternative to AWS S3 ● Decoupled from OpenStack ● Suitable for unstructured data ● Scales horizontally ● But you can't mount it! Logical Hierarchy Accounts/Containers/Objects 

gluster-swift ➔ Just another application using fuse mount ➔ Uses and “plugs into” OpenStack Swift ➔ Against having to write our own object front-end ➔ “Turns off” Swift’s distribution and replication 

Object access trivia (swift) ➔ Object = {Data + Metadata + Identifier} ➔ Abstracted ➔ Stateless (HTTP) ➔ No partial PUTs ➔ Ranged GETs supported ➔ No renames, only COPY ➔ Fixed logical hierarchy 

Vanilla Swift Ring SQLite DB SQLite DB XFS Account Server Container Server Object Server Proxy Server Client HTTP 

Pluggability in Swift • Backends implement DiskFile • Object server is pluggable officially. • Account and container code are not. • Makes cross-version compatibility tricky. 

Gluster-Swift Ring Account Server Container Server Object Server Proxy Server Client Same node FUSE mount 

Proxy Server - pipeline /etc/swift/proxy-server.conf pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server ➔ Very similar to translators in gluster ➔ Middlewares are highly modular ➔ Many features are implemented as middlewares 

Amazon S3 API Support ➔ Implemented as a middleware (swift3) ➔ Converts incoming S3 requests into Swift requests S3 Request Swift Response Swift3 middleware Swift Request S3 Response Proxy Server S3 Client 

Swift Object Ops → Fops PUT getxattr() mkdir()* open() write() fsetxattr() fsync() rename() close() GET open() fstat() fgetxattr() read() close() POST getxattr() stat() setxattr() DELETE getxattr() stat() unlink() rmdir()* *multiple calls - recursive HEAD getxattr() stat() 

# getfattr -d /mnt/gluster-object/test/c1/animals/cat/kitten user.swift.metadata="{ \"Content-Length\":\"4\", \"Etag\":\"4a4be40c96ac6314e91d93f38043a634\", \"X-Timestamp\":\"1475214280.33138\", \"X-Object-Type\":\"file\", \"X-Type\":\"Object\", \"Content-Type\":\"application/x-www-form-urlencoded\"}" # curl -i http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT -d'meow' HTTP/1.1 201 Created Last-Modified: Fri, 30 Sep 2016 05:44:41 GMT Content-Length: 0 Etag: 4a4be40c96ac6314e91d93f38043a634 Content-Type: text/html; charset=UTF-8 X-Trans-Id: txae858eb9169a4e1eb2a09-0057edfbc8 Date: Fri, 30 Sep 2016 05:44:40 GMT Object PUT 

● Atomicity ● open() with O_TMPFILE + linkat() ● mkstemp() and rename() That DHT trick that every user should know about: .filename.suffix hashed to same brick as filename .kitten.jpg.4e91d93f380 kitten.jpg → ● Multiple client PUTs ? ● Last write wins Object PUT 

Swift Listing Ops → Fops Container GET (plain text) – only names getxattr() opendir() readdir()* getxattr()1 Container GET (json/xml) – names + metadata getxattr() opendir() readdir()* getxattr()2 *multiple calls – recursive 1 only on dirs 2 on every inode 

Swift API Deviations # curl -i http://localhost:8080/v1/AUTH_test/c2 -X GET HTTP/1.1 200 OK Content-Length: 7 X-Container-Object-Count: 2 Accept-Ranges: bytes X-Storage-Policy: Policy-0 X-Container-Bytes-Used: 0 X-Timestamp: 0000000001.00000 Content-Type: text/plain; charset=utf-8 X-Trans-Id: tx18e21ccd03f04f4698a1a-0057ed1b84 Date: Thu, 29 Sep 2016 13:47:48 GMT HTTP Header needs to go first, then the response body. 

Swift API Deviations # curl http://host:8080/v1/AUTH_test/c1/animals/cat -X PUT Now, cat is a file # curl http://host:8080/v1/AUTH_test/c1/animals/cat/kitten -X PUT Here, cat is supposed to be directory but it’s already a file. Backend object path on filesystem gluster-swift: /mnt/gluster-object/test/c1/animals/cat Vanilla Swift: /mnt/sdb1/objects/778/69f/c2b307d78b6c419c0c1b76d91c08c69f/1412628708.01757.data 

➔ Zero copy and copy offload ➔ Use libgfapi python bindings ➔ Use hummingbird project ➔ Native object ? glfs_put(..), glfs_get(..) Ideas 

Demo ? https://asciinema.org/a/88026