Basics of Distributed Task Processing with Work Queues (AMQP)

A presentation I prepared for a seminar at my Institute. This idea will help professors at our Institute tap the potential of the available hardware to carry out faster computations for their research.
The idea is already common these days in web applications where it may not be possible to process a request as soon as it is received.
There are many patterns or models in the AMQP protocol of which Work Queues is one simple yet very useful model.

Transcript

1. Speeding up lengthy scientific computations with Distributed Task Processing. Pranjal Mittal B.Tech. P IV IIT B.H.U, Varanasi 10105EN028
2. Agenda • To start off with a practical problem scenario. • 2 naive ways to approach the scenario. • A smarter way to approach the problem with a Live Demonstration. • An animated look at the “Distributed Task Processing” model used in the demo.
3. Lets start with a problem. • Professor Mr. ECE has written a code, (a function) during his research that allows him to process an input speech signal and return useful results. • The function takes 10 seconds of processing time on an average. • But he has over 100 such input speech signals to process.
4. One “naive” solution 1. Open one machine 2. Pull your code (function) on it 3. Give your input to the function 4. Wait for 10 seconds or more for processing to be complete. 5. Repeat Step 3 & 4 hundred times for all inputs.
5. Was that very convenient? Total Time taken > 1000 Seconds
6. Second “naive” solution Install your code on 5 machines- A,B,C,D,E Divide input signal files to the 5 machines equally. Go to machine A. Provide input. ... Go to machine E. Provide input. Go to machine A again. Provide next input. ... A D C E B
7. Better? But was that still convenient?
8. Enter.. The idea of Work Queues, and Distributed Task Processing. Task 1 Task 2 Task 3
9. Lets see a live demo ..before we continue with an animation for the model • We will use integer inputs instead of an input signal. • We will accumulate the inputs into a task queue. • The processing function, termed as a “worker” • The worker will run on different machines and pick up tasks from the task queue.
10. • Queue the task onto the “Work Queue" Task 1 Task 2 Task 3 A Work Queue
11. Task 1 Task 2 Task 3 Task 4 Task Creator Adds Tasks to Work Queue • Queue the task onto the “Work Queue" Workers
12. Task 1 Task 2 Task 3 Task Creator Adds Task to Work Queue Worker 1 Task 4 Meanwhile, a free worker picks up the task already in the queue and becomes busy. • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
13. Task 2 Task 3 Task 4 Task Creator Adds Task to Work Queue Task 1 Task 5 One worker is busy performing a task. So the next worker picks up the other task. • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
14. Task 3 Task 4 Task 5 Task Creator Adds Task to Work Queue Task 1 Task 2 Now 2 workers are busy. Worker 1 might be about to finish his job. More tasks can be accumulated into the Message Queue. No worries - The queue can store a lot of data. • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
15. Task 4 Task 5 Task Creator Adds Task to Work Queue Task 2 Task 3 Worker 1 has finished his job and is ready to pick up the next job. It may happen that one task is smaller than the others. • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
16. Task 5 Task Creator Adds Task to Work Queue Task 4 • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
17. Task Creator Adds Task to Work Queue Task 5 • Queue the task onto the “Work Queue" • Workers pick and process a task when free.
18. Task Creator Adds Task to Work Queue • All tasks are solved when the task queue is finished.
19. Takeaway & Conclusion • The idea is to solve repetitive, computationally intensive problems using Distributed Task Queues • This idea is a part of the Advanced Message Queuing Protocol Standard - AMQP - widely adopted in modern day distributed systems. • Unlike other models for distributed processing this is very simple so it can be adopted by any researcher and professor without learning new difficult methods.
20. Thank you I would love to help get this model adopted at our department to make research computations become more efficient. PRANJAL MITTAL 10105EN028 B.Tech. P IV IIT BHU, Varanasi