OpenStack Swift 101 @ Open Source India 2016

Slides to talk presented at Open Source India 2016 (http://opensourceindia.in/osidays/) held at NIMHANS Convention Center, Bengaluru

Transcript

1. OpenStack Swift 101 Prashanth Pai Software Engineer Red Hat
2. 2 Types Of Storage BLOCK FILE OBJECT Low level I/O Sectors No metadata SCSI, SATA, ATA SAN Directories & Files NFS, Local FS Simplicity Rigid Metadata NAS Logical Hierarchy Unique Identifier Flexible Metadata Abstracted Stateless (HTTP) *Cinder *Manila *Swift ?
4. 4 OpenStack Swift ? • Alternative to Amazon S3 • Decoupled from other OpenStack components • Suitable for unstructured data • Scales horizontally • But you can't mount! Swift is a highly available, distributed, eventually consistent object store. Logical Hierarchy Accounts -> Containers -> Objects
5. 5 Object trivia (swift) ➔ Object = {Data + Metadata + Identifier} ➔ PUT = Upload; GET = Download ➔ No partial PUTs ➔ Ranged GETs supported ➔ No renames, only COPY
6. 6 Architecture Ring SQLite DB SQLite DB XFS Account Server Container Server Object Server Proxy Server Client HTTP
7. 7 The Swift API Each object’s access path consists of exactly three elements: /account/container/object GET 'X-Auth-Token: blah_blah' http://127.0.0.1:8080/v1/AUTH_test/c1/a/b/c/d/z Proxy server Account Container Object
8. 8 Middlewares /etc/swift/proxy-server.conf pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit tempauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server ➔ Proxy server pipeline ➔ Highly modular ➔ Features implemented as middlewares
9. 9 Amazon S3 API Support ➔ Implemented as a middleware (swift3) ➔ Converts incoming S3 requests into Swift requests S3 Request Swift Response Swift3 middleware Swift Request S3 Response Proxy Server S3 Client
10. 10 Data Placement Path = account/container/object Hash = md5sum( prefix + Path + suffix) Path = account/container/object Hash = md5sum( prefix + Path + suffix) • Based on hashing of object name • Eliminates need for central metadata server • No single point of failure
11. 11 https://www.openstack.org/assets/presentation-media/swiftpolicie.pdf • “Ring files” = Cluster map • Recreated when cluster layout changes • Hash range divided into partitions • Devices are assigned to partitions • Replica algorithm: As far as possible Data Placement – The Ring
12. 12 • Set on individual container • Applied to objects in that specific container Define policies based on... • Geography • Hardware properties • EC vs Replica • Backend Storage (like GlusterFS or Ceph) Storage Policies
13. 13 Thank You IRC : #openstack-swift