using Queue Service to batch processing

Transcript

1. using Queue Service
   for batch processing
   

   View Slide

2. Job Queue
   data source
   convert the data into job and sent to queue
   1
   

   View Slide

3. Job Queue
   data source
   1
   convert the data into job and sent to queue
   worker
   worker
   worker
   worker
   worker
   2
   pull job and consume it
   

   View Slide

4. Job Queue
   data source
   1
   convert the data into job and sent to queue
   worker
   worker
   worker
   worker
   worker
   2
   pull job and consume it
   Result Queue
   3
   send result to another queue
   

   View Slide

5. Job Queue
   data source
   1
   convert the data into job and sent to queue
   worker
   worker
   worker
   worker
   worker
   2
   pull job and consume it
   Result Queue
   3
   send result to another queue
   4
   feed back to data
   aggregator
   pull result
   

   View Slide

6. Job Queue
   data source
   1
   convert the data into job and sent to queue
   worker
   worker
   worker
   worker
   worker
   2
   pull job and consume it
   Result Queue
   3
   send result to another queue
   4
   aggregator
   pull result
   make report
   

   View Slide

7. Example
   for video resolution
   

   View Slide

8. Job Queue
   video table
   {id, url, how-to-parse}
   1
   

   View Slide

9. Job Queue
   1
   worker
   worker
   worker
   worker
   worker
   2
   pull job and consume it
   video table
   {id, url, how-to-parse}
   

   View Slide

10. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    {id, width, height, codec}
    video table
    {id, url, how-to-parse}
    

    View Slide

11. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    update video where id=?
    set width=?, height=?, codec=?
    video table
    {id, url, how-to-parse}
    {id, width, height, codec}
    4
    aggregator
    pull result
    

    View Slide

12. Example
    for streaming alive
    

    View Slide

13. Job Queue
    streaming table
    {id, url}
    1
    

    View Slide

14. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    streaming table
    {id, url}
    

    View Slide

15. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    {id, is_alive}
    streaming table
    {id, url}
    

    View Slide

16. Job Queue
    1
    worker
    worker
    worker
    worker
    worker
    2
    pull job and consume it
    Result Queue
    3
    update streaming_status
    where id=?
    set status=?
    4
    aggregator
    pull result
    streaming table
    {id, url}
    {id, is_alive}
    

    View Slide

17. Deployment
    

    View Slide

18. Job Queue
    worker
    worker
    worker
    worker
    worker
    Result Queue
    aggregator
    Everything can deploy locally,
    But
    

    View Slide

19. Job Queue
    worker
    worker
    worker
    worker
    worker
    Result Queue
    aggregator
    Deploy Workers on the Cloud
    let it more scalable
    

    View Slide