GlusterFS in OpenStack Prashanth Pai Red Hat, Bangalore GlusterFS Meetup, Pune Feb 7, 2015 

GlusterFS Meetup, Pune 2 OpenStack ? ● Started by NASA and Rackspace in 2010 ● Build your own IaaS cloud ● Modular and Plugg-able ● Similar to AWS, Azure, GCP ● OpenStack Foundation ● Written in Python ● 6 month release cadence 

GlusterFS Meetup, Pune 3 

GlusterFS Meetup, Pune 4 And much more... ● Telemetry (Ceilometer) ● Orchestration (Heat) ● Database (Trove) ● Bare Metal Provisioning (Ironic) ● Elastic Map Reduce (Sahara) ● Shared Filesystem As Service (Manila) ● TripleO 

GlusterFS Meetup, Pune 5 Storage in OpenStack* (and in general) BLOCK FILE OBJECT Low level I/O Tracks & Sectors No metadata SCSI, SATA, ATA SAN Directories & Files NFS, Local FS Simplicity NAS Rigid Metadata Containers / Buckets Data, Metadata & Identifier HTTP Abstracted *Cinder *Manila *Swift ? 

GlusterFS Meetup, Pune 6 Cinder ● Spun off from Nova ● Block storage as a service ● Similar to Amazon EBS ● Cinder volumes (block devices) are attached to VMs ● Volume lifecycle independant of VM instances ● Vendor specific backend drivers, LVM being the default ● Backend devices opaque to users ● Consistent API regardless of backend selection 

GlusterFS Meetup, Pune 7 Cinder API Cinder Scheduler DB Vendor Driver Vendor Driver Cinder Volume Cinder Backup STORAGE STORAGE ● Volume Types ● Quotas ● Migration ● Encryption ● Snapshot ● Extra Specs 

GlusterFS Meetup, Pune 8 

GlusterFS Meetup, Pune 9 Cinder - GlusterFS Integration ● Available as a backend driver in Cinder ● 1 Cinder volume == 1 GlusterFS file ● Support raw and qcow2 format cinder volumes ● GlusterFS backed cinder volumes can be attached to Nova instances in 2 ways – FUSE – Libgfapi ● Supports most of the Cinder APIs (including libgfapi based snapshots) ● cinder.conf - glusterfs_shares_config lists the glusterfs volumes to use in hostname:/volname format 

GlusterFS Meetup, Pune 10 Manila ● Shared FileSystem as a service ● Project created as fork of Cinder ● Create file shares to be consumed by Nova instances ● Also intended to be accessible as an independent service ● Part of the control plane and is only an orchestrator ● Vendor neutral API with support for multiple protocols and backend implementations. NFS and CIFS being default. ● Supports Multi-tenancy manila create [­­snapshot­id ] [­­name ] [­­description ] [­­volume­type ] manila list manila show manila delete 

GlusterFS Meetup, Pune 11 Manila usecase ● Share ● Share Protocol ● Share Server KEY TERMS ● Volume Types ● Snapshot ● Extra Specs KEY TERMS 

GlusterFS Meetup, Pune 12 GlusterFS Native Driver ● Supports Certificate based access type of Manila ● Provision shares that use the 'glusterfs' (FUSE) protocol ● Instances directly talk with GlusterFS storage backend – No service VM needed ● Secure access: Only tenants with the right certificate will be able to access the share ● Multi-tenant: Separation using tenant specific certificates ● Driver Available upstream – 1 Manila share == 1 GlusterFS volume – Pre-requisites ● GlusterFS volume(s) setup with Cert based access enabled ● Instance should have server signed client certificates pre-loaded ● Manila.conf – Provide list of glusterfs volume(s) to work with 

GlusterFS Meetup, Pune 13 GlusterFS Driver (NFS) ● Flat Network Driver ● A Manila Share maps to a subdirectory within a GlusterFS volume ● Shares exported by GlusterNFS (v3) residing in the GlusterFS storage pool GlusterFS Driver (Ganesha) ● WIP: Flat Network Ganesha + GlusterFS driver - NFSv3 and v4 - Modular architecture of Ganesha driver allows easy integration of GlusterFS, and other drivers ● Future: Network Segmented Ganesha + GlusterFS driver - will be a fully open source, multi-tenant Manila stack with a powerful distributed FS backend 

GlusterFS Meetup, Pune 14 OpenStack Swift ● Object is { Data + Metadata + Identifier } ● Alternative to Amazon S3 ● Decoupled/Independent from OpenStack ● Unstructured data ● Scales horizontally Swift is a highly available, distributed, eventually consistent object store. But you can't.. ● Mount it ● Have file hierarchies Hierarchy Account -> Containers -> Objects Hierarchy and Namespace Account -> Containers -> Objects 

GlusterFS Meetup, Pune 15 

GlusterFS Meetup, Pune 16 Rings ● Internal data structures stored in “ring files” in /etc/swift ● Maps objects to physical location on disk ● Uses hashing to place objects just like GlusterFS :) ● Eliminates need for central meta-data server ● Needs to be recreated when cluster layout changes Path = account/container/object Hash = md5sum( prefix + Path + suffix) 

GlusterFS Meetup, Pune 17 All operations are carried out by sending HTTP requests. Each object’s access path consists of exactly three elements: /account/container/object GET 'X-Auth-Token: blah_blah' http://example.com:8080/v1/AUTH_test/c1/a/b/c.jpg Proxy server Account Container Object Request method Request header 

GlusterFS Meetup, Pune 18 SwiftOnFile Project ● Allows objects created using Swift API to be accessed as files and vice-versa ● Map URL directly to file path ● Deployed as a Storage Policy ● Leverage distributed filesystem features ● Extend an existing Swift cluster you might already have ● Provide object interface to existing GlusterFS cluster Example Usecases Video transcoding, Analytics, Scientific Collaboration 

Thank You ppai on #gluster-dev, #swiftonfile, #openstack-swift