Using Redis to orchestrate cloud-based file transfers at scale

Using Redis to orchestrate cloud-based file transfers at scale

A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda

Presented at RedisConf19

E2e942d05d5ea93ad2eddd3394ffcd96?s=128

Carlos Justiniano

April 03, 2019
Tweet

Transcript

  1. None
  2. About me • Carlos Justiniano • VP of Engineering at

    Flywheel Sports • 2005 World Record in Distributed Computation • Leveraging the power of Redis since 2011 @cjus on Github, Twitter, Medium and at flywheelsports.com More about me at http://cjus.me
  3. What we’ll cover today A case study involving massive file

    transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda
  4. None
  5. None
  6. None
  7. None
  8. None
  9. Broadcasting live from NYC

  10. None
  11. None
  12. None
  13. Our video content transfer use case

  14. Challenges Quickly migrate our entire video library from one CDN

    to another: • Object Storage • HTTP Live Streaming (HLS) • Ensuring no file is left behind
  15. None
  16. Begins with a live broadcast Ends with an at-home rider

  17. Multiple manifests files pointing to collections of file segments Manifest

    file Segment files
  18. 91011213

  19. 91011213

  20. 91011213

  21. 91011213

  22. 91011213 91007688 91009060 91009473

  23. 91011213 91007688 91009060 91009473 91012341 91007687 91009062 91011625 91012265 91007686

    91007685 91007683 90999930 91009778 91011977 91011943
  24. 91009062

  25. 2000 classes 16 streams per class ~500 file* segments per

    stream *each file segment ranges from 100 bytes to 2 megabytes in size
  26. 2000 classes 16 streams per class ~500 file* segments per

    stream 2000 x 16 x 500 = 16,000,000 files *each file segment ranges from 100 bytes to 2 megabytes in size
  27. Our Solution • Pull individual files through the Verizon CDN

    • Web crawling manifest files • Use Redis powered Microservices to orchestrate millions of AWS Lambda invocations
  28. None
  29. Fly Live Ants

  30. None
  31. None
  32. Redis Messaging and Job Queuing

  33. None
  34. None
  35. Class scanner code

  36. None
  37. Crawler code

  38. None
  39. None
  40. Segment-transfer code

  41. None
  42. None
  43. None
  44. λ code

  45. None
  46. None
  47. Seeing the solution in action

  48. Completed segment In progress segment

  49. None
  50. End Results

  51. End results • The speed of transferring files using this

    approach is absolutely staggering. • During earlier tests, the system transferred four terabytes of data in two hours and twenty minutes! • That’s roughly 523MB per second! • Nowhere near the maximum potential. • Using both a larger multi-core or cluster of multi-core machines and a higher concurrent limit of lambda invocations would yield even higher transfer speeds.
  52. Lessons

  53. None
  54. None