using Queue Service to batch processing

Transcript

1. using Queue Service for batch processing
2. Job Queue data source convert the data into job and sent to queue 1
3. Job Queue data source 1 convert the data into job and sent to queue worker worker worker worker worker 2 pull job and consume it
4. Job Queue data source 1 convert the data into job and sent to queue worker worker worker worker worker 2 pull job and consume it Result Queue 3 send result to another queue
5. Job Queue data source 1 convert the data into job and sent to queue worker worker worker worker worker 2 pull job and consume it Result Queue 3 send result to another queue 4 feed back to data aggregator pull result
6. Job Queue data source 1 convert the data into job and sent to queue worker worker worker worker worker 2 pull job and consume it Result Queue 3 send result to another queue 4 aggregator pull result make report
7. Example for video resolution
8. Job Queue video table {id, url, how-to-parse} 1
9. Job Queue 1 worker worker worker worker worker 2 pull job and consume it video table {id, url, how-to-parse}
10. Job Queue 1 worker worker worker worker worker 2 pull job and consume it Result Queue 3 {id, width, height, codec} video table {id, url, how-to-parse}
11. Job Queue 1 worker worker worker worker worker 2 pull job and consume it Result Queue 3 update video where id=? set width=?, height=?, codec=? video table {id, url, how-to-parse} {id, width, height, codec} 4 aggregator pull result
12. Example for streaming alive
13. Job Queue streaming table {id, url} 1
14. Job Queue 1 worker worker worker worker worker 2 pull job and consume it streaming table {id, url}
15. Job Queue 1 worker worker worker worker worker 2 pull job and consume it Result Queue 3 {id, is_alive} streaming table {id, url}
16. Job Queue 1 worker worker worker worker worker 2 pull job and consume it Result Queue 3 update streaming_status where id=? set status=? 4 aggregator pull result streaming table {id, url} {id, is_alive}
17. Deployment
18. Job Queue worker worker worker worker worker Result Queue aggregator Everything can deploy locally, But
19. Job Queue worker worker worker worker worker Result Queue aggregator Deploy Workers on the Cloud let it more scalable