About me ● Carlos Justiniano ● VP of Engineering at Flywheel Sports ● 2005 World Record in Distributed Computation ● Leveraging the power of Redis since 2011 @cjus on Github, Twitter, Medium and at flywheelsports.com More about me at http://cjus.me
What we’ll cover today A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda
Challenges Quickly migrate our entire video library from one CDN to another: ● Object Storage ● HTTP Live Streaming (HLS) ● Ensuring no file is left behind
2000 classes 16 streams per class ~500 file* segments per stream 2000 x 16 x 500 = 16,000,000 files *each file segment ranges from 100 bytes to 2 megabytes in size
Our Solution ● Pull individual files through the Verizon CDN ● Web crawling manifest files ● Use Redis powered Microservices to orchestrate millions of AWS Lambda invocations
End results ● The speed of transferring files using this approach is absolutely staggering. ● During earlier tests, the system transferred four terabytes of data in two hours and twenty minutes! ● That’s roughly 523MB per second! ● Nowhere near the maximum potential. ● Using both a larger multi-core or cluster of multi-core machines and a higher concurrent limit of lambda invocations would yield even higher transfer speeds.