Moving to the Clouds for a Django site: Amazon S3 & Cloud Front vs. Rackspace Cloud Files & Akamai
Comparison of the 2 major cloud providers for storing and serving static files and the challenges for a Django site.
This is a presentation for LA Django meetup.
vs. Amazon Akamai s3 objects Tools End LA Django Nov 19 2013 Sep Dehpour www.linkedin.com/in/sepehr www.github.com/erasmose Moving to the Clouds: Amazon S3 & Cloud Front vs. Rackspace Cloud Files & Akamai Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Sep Dehpour is the primary engineer on a Django site that has over 1.5 million page views per day. His responsibilities range from systems administration, Chef DevOps, performance optimizations to new feature development. Sep is an avid skier and likes to think he is a good surfer too, but that shit ain't easy. About the presenter Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End In this talk: Comparison of the 2 major cloud providers for storing and serving sta1c files and the challenges for a Django site. Rackspace & Akamai vs. Amazon & CloudFront Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Comparison in glance Pay as you grow from 10¢/GB/Mo Akamai CDN at 12¢/GB at 213 PoP Store files and media of any size Highly scalable, redundant Pay as you grow from 9.5¢/GB/Mo Cloudfront CDN at 12¢/GB Store files and media of any size Highly scalable, redundant Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End CDN Comparison in glance Akamai is delivering 20-30% of web Apple, Netflix, Amazon.com 1,200 points of presence (PoP) But Rackspace plan uses 213 PoP Akamai NetSession Interface peer-to-peer Nasa, PBS 40 super PoP no peer-to-peer Tip: Use hQp://www.cdnplanet.com/tools/cdnfinder to find the CDN used to serve sta1c content. Put the full path to an image or CSS file to see what CDN is serving it. Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End So what’s exactly on S3/Cloud Files? Container{ Containers at the root level holding objects Object { “key”:”path to file to simulate a nested folder structure”, “etag”:MD5 checksum, “content_type”:Mime Type, “ttl”: time to expire on CDN }, ... } Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End CloudFuse S3fs Virtual File System Unlike tradi1onal file systems that essen1ally save data to and retrieve data from disk, virtual filesystems do not actually store data themselves. They act as a view or transla1on of an exis1ng file system or storage device. Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Buggy: CloudFuse S3fs (?) Virtual File System Unlike tradi1onal file systems that essen1ally save data to and retrieve data from disk, virtual filesystems do not actually store data themselves. They act as a view or transla1on of an exis1ng file system or storage device. Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End http://s3tools.org/s3cmd Run s3cmd --configure Run s3cmd ls to list all your buckets. Make a bucket with s3cmd mb s3://my-new-bucket-name List the contents of the bucket s3cmd ls s3://bucket_name Upload a file into the bucket s3cmd put addressbook.xml s3://logix.cz-test/addrbook.xml Command line S3cmd Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End import pyrax pyrax.set_credential_file("/home/user/.pyrax.cfg") cf = pyrax.cloudfiles print "list of containers:", cf.list_containers() cont = cf.get_container("my_container") #uploading a folder local = "path to a local folder" cf.sync_folder_to_container(folder_path=local, container=cont, include_hidden=False) Command line Pyrax Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End from boto.s3.connection import S3Connection from boto.s3.key import Key import sys conn = S3Connection('AWS credentials', 'AWS credentials') bucket = conn.get_bucket('bucketname') rs = bucket.list("Keys starting with this string") for key in rs: print key.name Command line Boto Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Django (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Always use Django default_storage so you can easily switch the back-end storage from django.core.files.storage import default_storage from django.core.files.base import ContentFile path = default_storage.save('/path/to/file', ContentFile('new content')) path u'/path/to/file' default_storage.size(path) 11 default_storage.open(path).read() 'new content' default_storage.delete(path) default_storage.exists(path) False Django Django Storages Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End But the 3rd party app that we were using did custom file writing instead of using Django default_storage Django Django Storages Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Toolsets/libraries Rackspace control panel AWS control panel Web Toolkit Swift Client (written in Python) S3cmd (written in Python) Command line CloudFuse S3fs Virtual File System Pyrax Boto Python Django Cumulus Django Storages Django Go s3 Python (other) Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End Go S3: A light wrapper around Python Boto https://github.com/erasmose/go_s3 When to use Go_s3? You have custom file read/write in your apps or 3rd party apps that are not compatible with “Django Storages” and you don’t have time to go through making it use “Django default storage” so it gets compatible with “Django Storages” or You want to have full control over what goes on S3 and its access control (ACL) Python Go S3 Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End ACL canned access control policy a. private: Owner gets FULL_CONTROL. No one else has any access rights. b. public-‐read: Owners gets FULL_CONTROL and the anonymous principal is granted READ access. c. public-‐read-‐write: Owner gets FULL_CONTROL and the anonymous principal is granted READ and WRITE access. d. authenticated-‐read: Owner gets FULL_CONTROL and any principal authenticated as a registered Amazon S3 user is granted READ access Python Go S3 Wednesday, November 20, 13
vs. Amazon Akamai s3 objects Tools End LA Django Nov 19 2013 Sep Dehpour www.linkedin.com/in/sepehr www.github.com/erasmose Thanks to TrueCar for hos1ng Wednesday, November 20, 13
champ
faithandbrave
z63d
fujisawaryohei
miichan
iidaxs
hirotomotaguchi
lycorptech_jp
fujiwara3
qt_luigi
mikanichinose
oguressive
morganepeng
bluesmoon
palkan
jnunemaker
lemiorhan
brad_frost
bkeepers
notwaldorf
kevinliebholz
orderedlist
panda_program
mza
