Upgrade to Pro — share decks privately, control downloads, hide ads and more …

using Queue Service to batch processing

using Queue Service to batch processing

Ching Yi Chan

October 08, 2013
Tweet

More Decks by Ching Yi Chan

Other Decks in How-to & DIY

Transcript

  1. using Queue Service
    for batch processing

    View Slide

  2. Job Queue
    data source
    convert the data into job and sent to queue
    1

    View Slide

  3. Job Queue
    data source
    1
    convert the data into job and sent to queue
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it

    View Slide

  4. Job Queue
    data source
    1
    convert the data into job and sent to queue
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    send result to another queue

    View Slide

  5. Job Queue
    data source
    1
    convert the data into job and sent to queue
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    send result to another queue
    4
    feed back to data
    aggregator
    pull result

    View Slide

  6. Job Queue
    data source
    1
    convert the data into job and sent to queue
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    send result to another queue
    4
    aggregator
    pull result
    make report

    View Slide

  7. Example
    for video resolution

    View Slide

  8. Job Queue
    video table
    {id, url, how-to-parse}
    1

    View Slide

  9. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    video table
    {id, url, how-to-parse}

    View Slide

  10. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    {id, width, height, codec}
    video table
    {id, url, how-to-parse}

    View Slide

  11. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    update video where id=?
    set width=?, height=?, codec=?
    video table
    {id, url, how-to-parse}
    {id, width, height, codec}
    4
    aggregator
    pull result

    View Slide

  12. Example
    for streaming alive

    View Slide

  13. Job Queue
    streaming table
    {id, url}
    1

    View Slide

  14. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    streaming table
    {id, url}

    View Slide

  15. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    {id, is_alive}
    streaming table
    {id, url}

    View Slide

  16. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    update streaming_status
    where id=?
    set status=?
    4
    aggregator
    pull result
    streaming table
    {id, url}
    {id, is_alive}

    View Slide

  17. Deployment

    View Slide

  18. Job Queue
    worker
    worker
    worker
    worker
    worker
    Result Queue
    aggregator
    Everything can deploy locally,
    But

    View Slide

  19. Job Queue
    worker
    worker
    worker
    worker
    worker
    Result Queue
    aggregator
    Deploy Workers on the Cloud
    let it more scalable

    View Slide