Using Redis to orchestrate cloud-based file transfers at scale

A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda

Presented at RedisConf19

Transcript

2. About me • Carlos Justiniano • VP of Engineering at Flywheel Sports • 2005 World Record in Distributed Computation • Leveraging the power of Redis since 2011 @cjus on Github, Twitter, Medium and at flywheelsports.com More about me at http://cjus.me
3. What we’ll cover today A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda
9. Broadcasting live from NYC
13. Our video content transfer use case
14. Challenges Quickly migrate our entire video library from one CDN to another: • Object Storage • HTTP Live Streaming (HLS) • Ensuring no file is left behind
16. Begins with a live broadcast Ends with an at-home rider
17. Multiple manifests files pointing to collections of file segments Manifest file Segment files
22. 91011213 91007688 91009060 91009473
23. 91011213 91007688 91009060 91009473 91012341 91007687 91009062 91011625 91012265 91007686 91007685 91007683 90999930 91009778 91011977 91011943
25. 2000 classes 16 streams per class ~500 file* segments per stream *each file segment ranges from 100 bytes to 2 megabytes in size
26. 2000 classes 16 streams per class ~500 file* segments per stream 2000 x 16 x 500 = 16,000,000 files *each file segment ranges from 100 bytes to 2 megabytes in size
27. Our Solution • Pull individual files through the Verizon CDN • Web crawling manifest files • Use Redis powered Microservices to orchestrate millions of AWS Lambda invocations
29. Fly Live Ants
32. Redis Messaging and Job Queuing
35. Class scanner code
37. Crawler code
40. Segment-transfer code
44. λ code
47. Seeing the solution in action
48. Completed segment In progress segment
50. End Results
51. End results • The speed of transferring files using this approach is absolutely staggering. • During earlier tests, the system transferred four terabytes of data in two hours and twenty minutes! • That’s roughly 523MB per second! • Nowhere near the maximum potential. • Using both a larger multi-core or cluster of multi-core machines and a higher concurrent limit of lambda invocations would yield even higher transfer speeds.
52. Lessons