Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Redis to orchestrate cloud-based file transfers at scale

Using Redis to orchestrate cloud-based file transfers at scale

A case study involving massive file transfers, Redis, Microservices, job creation and orchestration and Serverless computing using AWS Lambda

Presented at RedisConf19

Carlos Justiniano

April 03, 2019
Tweet

More Decks by Carlos Justiniano

Other Decks in Technology

Transcript

  1. View Slide

  2. About me
    ● Carlos Justiniano
    ● VP of Engineering at Flywheel Sports
    ● 2005 World Record in Distributed
    Computation
    ● Leveraging the power of Redis since 2011
    @cjus on Github, Twitter, Medium and at flywheelsports.com
    More about me at http://cjus.me

    View Slide

  3. What we’ll cover today
    A case study involving massive file
    transfers, Redis, Microservices, job
    creation and orchestration and
    Serverless computing using AWS
    Lambda

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. Broadcasting
    live
    from NYC

    View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. Our video content
    transfer use case

    View Slide

  14. Challenges
    Quickly migrate our entire video
    library from one CDN to another:
    ● Object Storage
    ● HTTP Live Streaming (HLS)
    ● Ensuring no file is left behind

    View Slide

  15. View Slide

  16. Begins with a live broadcast
    Ends with an at-home rider

    View Slide

  17. Multiple manifests files
    pointing to collections
    of file segments
    Manifest file Segment files

    View Slide

  18. 91011213

    View Slide

  19. 91011213

    View Slide

  20. 91011213

    View Slide

  21. 91011213

    View Slide

  22. 91011213 91007688
    91009060 91009473

    View Slide

  23. 91011213 91007688 91009060 91009473
    91012341 91007687 91009062 91011625
    91012265 91007686 91007685 91007683
    90999930 91009778 91011977 91011943

    View Slide

  24. 91009062

    View Slide

  25. 2000 classes
    16 streams per class
    ~500 file* segments per stream
    *each file segment ranges from
    100 bytes to 2 megabytes in
    size

    View Slide

  26. 2000 classes
    16 streams per class
    ~500 file* segments per stream
    2000 x 16 x 500 =
    16,000,000 files
    *each file segment ranges from
    100 bytes to 2 megabytes in
    size

    View Slide

  27. Our Solution
    ● Pull individual files through the
    Verizon CDN
    ● Web crawling manifest files
    ● Use Redis powered Microservices
    to orchestrate millions of AWS
    Lambda invocations

    View Slide

  28. View Slide

  29. Fly Live Ants

    View Slide

  30. View Slide

  31. View Slide

  32. Redis Messaging and
    Job Queuing

    View Slide

  33. View Slide

  34. View Slide

  35. Class scanner code

    View Slide

  36. View Slide

  37. Crawler code

    View Slide

  38. View Slide

  39. View Slide

  40. Segment-transfer code

    View Slide

  41. View Slide

  42. View Slide

  43. View Slide

  44. λ code

    View Slide

  45. View Slide

  46. View Slide

  47. Seeing the solution
    in action

    View Slide

  48. Completed segment
    In progress segment

    View Slide

  49. View Slide

  50. End Results

    View Slide

  51. End results
    ● The speed of transferring files
    using this approach is absolutely
    staggering.
    ● During earlier tests, the system
    transferred four terabytes of data
    in two hours and twenty minutes!
    ● That’s roughly 523MB per
    second!
    ● Nowhere near the maximum
    potential.
    ● Using both a larger multi-core or
    cluster of multi-core machines
    and a higher concurrent limit of
    lambda invocations would yield
    even higher transfer speeds.

    View Slide

  52. Lessons

    View Slide

  53. View Slide

  54. View Slide