Using AWS CloudFront with S3 at SMARTSTUDY

AWS Korean User Group #1 Seminar at 2013-01

Transcript

1. Using CloudFront with S3 at SMARTSTUDY Amazon Web Service Korean User Group Park Hyun-woo, a SCV of SMARTSTUDY / ez@smartstudy.co.kr
2. What is S3 and CF? • S3 : Simple Storage Service ◦ Extreme durability. ◦ Multiple copies in selected region. ▪ vs EBS : Multiple copies in single available zone. • CF : CloudFront ◦ Content Delivery Network(CDN) service. ◦ Distribute from origin to AWS Edge Network ▪ http://aws.amazon.com/cloudfront/#details ▪ About 30 edges in global. (only in Earth, lol)
3. Why choose these? • We serve large multimedia contents to global market in mobile application era. • We need redundant, fast and reasonable price content delivery service aims globally. • We have not enough human resource to maintain various system.
4. Flash Movie
5. S3 Bucket ( tokyo ) s3cmd Package ( encrypted ) Thumbnails Video npk Transcoder Gnash FFmpeg qtfaststart Flash Movie
6. CloudFront { Edge Edge Edge Edge Edge S3 Bucket ( tokyo ) s3cmd Video npk
7. How it works? • Copy to S3 ◦ ~100 files, just use Cyberduck or other GUI tool. ◦ Or, you have to use s3fs or s3cmd. ◦ If you need sync or invalidation, use s3cmd. ◦ We have ~30,000 files. Thus, s3cmd. ◦ We chose Tokyo bucket for faster uploading. • s3cmd ◦ Awesome! ◦ Works on S3 and CloudFront.
8. How it works? (cont.) • s3cmd ◦ Can do almost everything as like as AWS console. ◦ Support INI-style configuration file. ◦ And, magical 'sync' command. (yeah!) • s3cmd sync ◦ Support glob-style exclude / include option. ◦ ... rexclude / rinclude for regular expression. ◦ Support automatic invalidation request. ◦ ... with rsync-like 'dry-run' option. (-n, --dry-run)
9. • s3cmd sync ◦ Basically, sync based on MD5 information. ◦ (!) Over 15MB files, using only filesize. ◦ Our 30,000 files, 60GB = 8 min to dry-run. ( env : i7 2500K + SATA HDD to bucket in tokyo ) • s3cmd sync --cf-invalidate ◦ Call invalidation API automatically. ◦ (!) Simultaneous invalidation is up to 3. ◦ (!) Free for ~1,000 files per month. ◦ Check it up by s3cmd cfinvalinfo How it works? (cont.)
10. So, is it really? FAST REDUNDANT NOT EXPENSIVE {
11. It's fast enough. 8.0MB/s
12. It's very redundant. BETTER THAN YOUR OWN SERVER, EVER
13. And NOT EXPENSIVE
14. * without transfer fee for deploying origin bucket to edges Bill for CloudFront Based on Total traffic • 10TB/month • $0.209/GB = $2,090* Bill for Domestic CDN Based on Peak traffic • 350Mbps/@peak • $6.00/Mbps = $2,100
15. STILL EXPENSIVE?
16. * Photo from Linkedin page of Lee, Jung-in / Amazon Korea
17. For over 10TB, RESERVED Capacity Pricing
18. • Smaller ◦ Encode media with optimal options. ◦ Use dedicated option for each platform. ◦ Storage is cheaper than transfer fee! • Static ◦ Do not deploy volatile objects to CloudFront. ◦ Invalidation needs ~10min, often. ◦ We're still using Nginx for them. • Streamable ◦ Mobile networks are still slow. ◦ Do not make users to wait until finishing download. Tips
19. • Gnash ◦ http://www.gnu.org/software/gnash/ • FFmpeg ◦ http://www.ffmpeg.org/ • qtfaststart ◦ https://github.com/danielgtaylor/qtfaststart • npk ◦ https://github.com/lqez/npk • s3cmd ◦ http://s3tools.org/ Link
20. Using CloudFront with S3 at SMARTSTUDY Amazon Web Service Korean User Group Park Hyun-woo, a SCV of SMARTSTUDY / ez@smartstudy.co.kr