Python bindings to GlusterFS - a distributed filesystem

Open Space Talk @ PyCon India 2016, New Delhi

https://in.pycon.org/cfp/open-space-2016/proposals/python-bindings-to-glusterfs-a-distributed-filesystem~dy6ze/

Transcript

1. Python bindings to GlusterFS - a distributed filesystem Open Space @ PyCon India 2016 September 24-25, New Delhi Prashanth Pai Software Engineer, Red Hat
2. ➔ Open source ➔ Distributed and scale out ➔ Highly available (failover) ➔ Single global namespace ➔ Commodity hardware ➔ Massively scalable GlusterFS
3. ➔ GlusterFS Native Client (FUSE) ➔ NFS ➔ SMB/CIFS ➔ Object (Swift and S3 APIs over HTTP) ➔ libgfapi library Data Access Methods
4. Node 1 FUSE NFS SMB Object Aggregated and presented as single namespace Node 2 Node 3 Node 4 Volume (2 x 2) (Distributed - Replicate) Cluster Architecture
5. Data Placement ➔ Uses hashing ➔ Hash ranges assigned to dirs ➔ Renames are special ➔ No metadata server ➔ Replication is synchronous ➔ Self healing (file repair) ➔ Optionally enforce quorum AFR /animals/cats/kitten.jpg 41 = hash_fn(kitten.jpg) DHT 0-49 0-49 AFR 50-99 50-99
6. FUSE: Filesystem in User Space https://en.wikipedia.org/wiki/Filesystem_in_Userspace#/media/File:FUSE_structure.svg CC BY-SA 3.0
7. Demo GlusterFS over FUSE
8. Image courtesy: https://website-humblec.rhcloud.com/wp-content/uploads/2014/04/screen2.jpg
9. ➔ Userspace C library ➔ Applications link against libgfapi.so ➔ Reduced context switching ➔ Extend and break-free from POSIX ? libgfapi
10. Image courtesy: https://website-humblec.rhcloud.com/wp-content/uploads/2014/04/screen1.jpg
11. ➔ Implemented using ctypes ➔ APIs mimic I/O methods of ‘os’ module ➔ But no monkey patching ➔ Available over PyPI ➔ Existing applications need code changes ➔ No Py3 support, yet libgfapi-python
12. Example and Demo libgfapi-python
13. # Create virtual mount v = Volume('10.7.1.99', 'data') v.mount() # Create directory v.mkdir('/dir1') # List directories v.listdir('/') # Create new file and write to it with v.fopen('/somefile.txt', 'w') as f: f.write("Winter is coming.") # Open and read file with v.fopen('/somefile.txt', 'r') as f: print f.read() # Delete file v.unlink('/somefile.txt') # Unmount v.umount() # Create directory os.mkdir('/dir1') # List directories os.listdir('/') # Create new file and write to it with open('/somefile.txt', 'w') as f: f.write("Winter is coming.") # Open and read file with open('/somefile.txt', 'r') as f: print f.read() # Delete file os.unlink('/somefile.txt')
14. Documentation and API Reference: http://libgfapi-python.rtfd.io Source: https://github.com/gluster/libgfapi-python Reach out to Gluster community: #gluster-dev on freenode gluster-devel@gluster.org