Python bindings to GlusterFS - a distributed filesystem

Faf5e6dafbdb19ab1dccaa1035ef3ce8?s=47 Prashanth Pai
September 25, 2016

Python bindings to GlusterFS - a distributed filesystem

Faf5e6dafbdb19ab1dccaa1035ef3ce8?s=128

Prashanth Pai

September 25, 2016
Tweet

Transcript

  1. Python bindings to GlusterFS - a distributed filesystem Open Space

    @ PyCon India 2016 September 24-25, New Delhi Prashanth Pai Software Engineer, Red Hat
  2. ➔ Open source ➔ Distributed and scale out ➔ Highly

    available (failover) ➔ Single global namespace ➔ Commodity hardware ➔ Massively scalable GlusterFS
  3. ➔ GlusterFS Native Client (FUSE) ➔ NFS ➔ SMB/CIFS ➔

    Object (Swift and S3 APIs over HTTP) ➔ libgfapi library Data Access Methods
  4. Node 1 FUSE NFS SMB Object Aggregated and presented as

    single namespace Node 2 Node 3 Node 4 Volume (2 x 2) (Distributed - Replicate) Cluster Architecture
  5. Data Placement ➔ Uses hashing ➔ Hash ranges assigned to

    dirs ➔ Renames are special ➔ No metadata server ➔ Replication is synchronous ➔ Self healing (file repair) ➔ Optionally enforce quorum AFR /animals/cats/kitten.jpg 41 = hash_fn(kitten.jpg) DHT 0-49 0-49 AFR 50-99 50-99
  6. FUSE: Filesystem in User Space https://en.wikipedia.org/wiki/Filesystem_in_Userspace#/media/File:FUSE_structure.svg CC BY-SA 3.0

  7. Demo GlusterFS over FUSE

  8. Image courtesy: https://website-humblec.rhcloud.com/wp-content/uploads/2014/04/screen2.jpg

  9. ➔ Userspace C library ➔ Applications link against libgfapi.so ➔

    Reduced context switching ➔ Extend and break-free from POSIX ? libgfapi
  10. Image courtesy: https://website-humblec.rhcloud.com/wp-content/uploads/2014/04/screen1.jpg

  11. ➔ Implemented using ctypes ➔ APIs mimic I/O methods of

    ‘os’ module ➔ But no monkey patching ➔ Available over PyPI ➔ Existing applications need code changes ➔ No Py3 support, yet libgfapi-python
  12. Example and Demo libgfapi-python

  13. # Create virtual mount v = Volume('10.7.1.99', 'data') v.mount() #

    Create directory v.mkdir('/dir1') # List directories v.listdir('/') # Create new file and write to it with v.fopen('/somefile.txt', 'w') as f: f.write("Winter is coming.") # Open and read file with v.fopen('/somefile.txt', 'r') as f: print f.read() # Delete file v.unlink('/somefile.txt') # Unmount v.umount() # Create directory os.mkdir('/dir1') # List directories os.listdir('/') # Create new file and write to it with open('/somefile.txt', 'w') as f: f.write("Winter is coming.") # Open and read file with open('/somefile.txt', 'r') as f: print f.read() # Delete file os.unlink('/somefile.txt')
  14. Documentation and API Reference: http://libgfapi-python.rtfd.io Source: https://github.com/gluster/libgfapi-python Reach out to

    Gluster community: #gluster-dev on freenode gluster-devel@gluster.org