Using Redis to orchestrate cloud-based file transfers at scale

A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda

Presented at RedisConf19

Transcript

1. View Slide

2. About me
   ● Carlos Justiniano
   ● VP of Engineering at Flywheel Sports
   ● 2005 World Record in Distributed
   Computation
   ● Leveraging the power of Redis since 2011
   @cjus on Github, Twitter, Medium and at flywheelsports.com
   More about me at http://cjus.me
   

   View Slide

3. What we’ll cover today
   A case study involving massive file
   transfers, Redis, Microservices, job
   creation and orchestration and
   Serverless computing using AWS
   Lambda
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. Broadcasting
   live
   from NYC
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. Our video content
    transfer use case
    

    View Slide

14. Challenges
    Quickly migrate our entire video
    library from one CDN to another:
    ● Object Storage
    ● HTTP Live Streaming (HLS)
    ● Ensuring no file is left behind
    

    View Slide

15. View Slide

16. Begins with a live broadcast
    Ends with an at-home rider
    

    View Slide

17. Multiple manifests files
    pointing to collections
    of file segments
    Manifest file Segment files
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. 91011213 91007688
    91009060 91009473
    

    View Slide

23. 91011213 91007688 91009060 91009473
    91012341 91007687 91009062 91011625
    91012265 91007686 91007685 91007683
    90999930 91009778 91011977 91011943
    

    View Slide

24. View Slide

25. 2000 classes
    16 streams per class
    ~500 file* segments per stream
    *each file segment ranges from
    100 bytes to 2 megabytes in
    size
    

    View Slide

26. 2000 classes
    16 streams per class
    ~500 file* segments per stream
    2000 x 16 x 500 =
    16,000,000 files
    *each file segment ranges from
    100 bytes to 2 megabytes in
    size
    

    View Slide

27. Our Solution
    ● Pull individual files through the
    Verizon CDN
    ● Web crawling manifest files
    ● Use Redis powered Microservices
    to orchestrate millions of AWS
    Lambda invocations
    

    View Slide

28. View Slide

29. Fly Live Ants
    

    View Slide

30. View Slide

31. View Slide

32. Redis Messaging and
    Job Queuing
    

    View Slide

33. View Slide

34. View Slide

35. Class scanner code
    

    View Slide

36. View Slide

37. Crawler code
    

    View Slide

38. View Slide

39. View Slide

40. Segment-transfer code
    

    View Slide

41. View Slide

42. View Slide

43. View Slide

44. λ code
    

    View Slide

45. View Slide

46. View Slide

47. Seeing the solution
    in action
    

    View Slide

48. Completed segment
    In progress segment
    

    View Slide

49. View Slide

50. End Results
    

    View Slide

51. End results
    ● The speed of transferring files
    using this approach is absolutely
    staggering.
    ● During earlier tests, the system
    transferred four terabytes of data
    in two hours and twenty minutes!
    ● That’s roughly 523MB per
    second!
    ● Nowhere near the maximum
    potential.
    ● Using both a larger multi-core or
    cluster of multi-core machines
    and a higher concurrent limit of
    lambda invocations would yield
    even higher transfer speeds.
    

    View Slide

52. Lessons
    

    View Slide

53. View Slide

54. View Slide