Upstream consultancy and Ceph RadosGW/S3

Ceph Day Santiago de Compostela
April 2018 @ CDTIC Xunta de Galicia, Santiago de Compostela (Spain)

https://cdtic.xunta.gal/ceph

Transcript

1. Upstream consultancy and Ceph RadosGW/S3 @javimunhoz
2. 2 Who? • Javier • Computer Engineer and Software Developer • Cloud and Virtualization team @ Igalia @javimunhoz linkedin.com/in/javiermunhoz
3. 3 This talk • Context and digital playground • Cloud Object Storage with Ceph RadosGW/S3 • Upstream consultancy examples • Q+A
4. 4 www.igalia.com
5. 5 Open Source software development • Contribution and Collaboration • Upstream, Backport, Fork ... • Patch, Release, Version, Feature, Bug, Fix … https://goo.gl/nPBVso
6. 6 Context and digital playground
7. 7 Digital products and services • The economics of connections – Give access to your value – Take advantage of new resources – Multiply the connections • The API economy – API as enabler for turning a business or organization into a platform – API expose assets like data, algorithms and transactions • The Data economy – Data is now a form of capital – Generate new digital product and services – New computing architecture based on services that are easier to buy and use
8. 8 Digital Markets https://ec.europa.eu/commission/priorities/digital-single-market_en
9. 9 Digital Markets https://ec.europa.eu/digital-single-market/en/news/fnal-results-european-data-market-study-measuring-size-and-trends-eu-data-economy
10. 10 The traditional company
11. 11 The company as platform
12. 12 BBVA API market ... https://www.bbvaapimarket.com/products
13. 13 BBVA API market … and data https://www.bbvaapimarket.com/products
14. 14 Cloud Object Storage with Ceph RadosGW/S3
15. 15 Ceph RadosGW/S3 demo https://goo.gl/cQKsSf
16. 16 Object Storage • Object – Data – Metadata – Global ID • Flat namespace – Bucket/Container – Objects • Objects are immutable
17. 17 Ceph RADOS cluster
18. 18 Cloud Object Storage • Web API • Protocol – Amazon S3 – OpenStack Swift – Google cloud storage – Azure blob storage • RadosGW
19. 19 Ceph RadosGW
20. 20 Amazon S3
21. 21 Features • Authentication • Static website hosting • Logging • Object versioning • Lifecycle management • Cross-region replication • Requester pays • Object torrent • Tagging • ... https://goo.gl/RMYuhU
22. 22 Features • Authentication • Static website hosting • Logging • Object versioning • Lifecycle management • Cross-region replication • Requester pays • Object torrent • Tagging • ... https://goo.gl/RMYuhU
23. 23 S3 example: GET bucket (List Objects) Version 2 GET /?list-type=2 HTTP/1.1 Host: bucket.s3.amazonaws.com x-amz-date: 20160430T233541Z Authorization: authorization string Content-Type: text/plain <?xml version="1.0" encoding="UTF-8"?> <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>bucket</Name> <Prefix/> <KeyCount>205</KeyCount> <MaxKeys>1000</MaxKeys> <IsTruncated>false</IsTruncated> <Contents> <Key>my-image.jpg</Key> <LastModified>2009-10-12T17:50:30.000Z</LastModified> <ETag>&quot;fba9dede5f27731c9771645a39863328&quot;</ETag> <Size>434234</Size> <StorageClass>STANDARD</StorageClass> </Contents> <Contents> ... </Contents> ... </ListBucketResult>
24. 24 Amazon S3 pricing • Storage pricing – Prices are based on the location (region) of the bucket – Storage classes and availability • Request pricing • Storage management – Inventory, analytic and object tagging • Transfer acceleration • Data transfer https://aws.amazon.com/s3/pricing/
25. 25 Ceph RadosGW/S3
26. 26 Ceph RadosGW internals
27. 27 Web Scale Object Storage A A
28. 28 Upstream consultancy examples
29. 29 www.outscale.com
30. 30 AWS Signature Version 4
31. 31 Requester Pay Bucket
32. 32 Multipart Upload (Copy part)
33. 33 Libcloud Storage Drivers
34. 34 Ansible AWS S3 core module
35. 35 Minio Cloud client bugfx rgw s3 auth aws4 force boto2 compat = false
36. 36 Community • ceph.com • ceph.com/community • ceph.com/get-involved @ceph Cephalocon 2018 APAC
37. Thanks! Q+A @javimunhoz