Applied AI: Refining an operational policy of a manufacturing system

A guest lecture in MDH given on 16th September, 2019

Transcript

1. Visualization in Industrial 4.0 Applied AI: Refining an operational policy of a manufacturing system Sep. 16th, 2019 Hajime Mizuyama mizuyama@ise.aoyama.ac.jp, hajimizu@kth.se 1
2. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 2
3. Visualization in Industrial 4.0 What do you see in a factory? Raw materials, Parts, Auxiliary materials, Half-made products, Finished products, etc. Human workers, Foremen, Engineers, etc. https://www.youtube.com/watc h?v=rt65167tZlQ Machine tools, Industrial robots, Automated guided vehicles, Automated warehouses, Conveyers, Cranes, etc. 3 Materials Men Machines Resources
4. Visualization in Industrial 4.0 A manufacturing system is … • A manufacturing system, or a factory, is a huge container of materials. • However, they are not simply kept in the system, but being transformed from raw materials to finished products step by step. • So, a manufacturing system is also a huge human-machine working system composed of various resources. • The resources are carrying out various steps of the whole transformation process of materials, which we call operations. • From a functional point of view, the system can also be conceptualized as a collection of those transforming operations. 4
5. Visualization in Industrial 4.0 Typical operations • Change locations (i.e. material handling) • Change how materials are grouped (lot/batch formations, etc.) • Change physical characteristics (shape, color, temperature, etc.) • Assemble/decompose • Add information (measurements, inspection, etc.) • Setup operations (changing tools, dies, machine settings, etc.) • Move empty containers, AGVs, etc. 5
6. Visualization in Industrial 4.0 Some characteristics of operations • Operations can be classified into value-added and non-value-added ones. – Operations are deemed value-added if they are indispensable for transforming raw materials into final products. • Most operations are carried out on materials, but some are not. • However, even some operations performed on materials can be non-value-added (changing locations, lot/batch formations, etc.). • Carrying out an operation takes time, and requires necessary resources (and materials). 6
7. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 7
8. Visualization in Industrial 4.0 Operating a manufacturing system is … • It is to have its resources carry out operations so that raw materials are transformed into finished products in an organized manner. • To do this, it needs to be determined what operations should be performed when by which resources (on which materials). • From production and operations management point of view, it is this decision that is essential and critical, because it affects: – Production lead-time, – Utilization of resources, – Inventory levels, – Amount of non-value added operations, etc. 8
9. Visualization in Industrial 4.0 Ex-post and ex-ante production schedule • The result can be shown and evaluated with the ex-post production schedule, which shows what operations were actually performed when by which resources (on which materials). • This ex-post schedule is only available retrospectively, and cannot be fully designed or optimized beforehand as an ex-ante schedule. – Uncertainties (forecast distributions vs. realized values) – Situatedness and locality (central control vs. autonomousness) • Central scheduling also often suffers from: – Combinatorial explosion – Incompatibility with structural changes/improvements (Kaizen) 9
10. Visualization in Industrial 4.0 How operational decisions are made Centralized system- wide planning Local distributed decisions made autonomously Offline planning made in advance based on forecast distribution of uncertainties Central/offline planning Online real-time decisions made when necessary based on the actual realized situation Local/online decisions 10
11. Visualization in Industrial 4.0 Practical operational framework 11 Central/offline planning Local/online decisions Goals, constraints, other system-wide information Actual progress, other situational information Real-time autonomous operational policies Rough-sketch system- wide optimization
12. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 12
13. Visualization in Industrial 4.0 Example #1: Inventory control 13 Supply Order Demand Shortage $ Ordering cost $ Holding cost $ Shortage penalty How many items should be ordered and when? Back orders Replenishment Lead time
14. Visualization in Industrial 4.0 Example #2: Dispatching in a job shop 14 J7 J6 J5 J9 J8 J11 J10 J3 J12 J4 J1 J2 When a machine becomes vacant, which job should be loaded on the machine next?
15. Visualization in Industrial 4.0 Example #3: Routing AGVs https://www.youtube.com/watch?v=4DKrcpa8Z_E 15
16. Visualization in Industrial 4.0 Example #4: LIFO buffers and a Crane Several kinds of jobs arrive randomly, and a due date is assigned to each of them. A setup operation is necessary, when changing job kinds to be processed on the machine. The jobs can be moved with a crane from the entrance to ,and between multiple LIFO buffers and a machine. LIFO: last in first out 16
17. Visualization in Industrial 4.0 A model of operational policy 17 Non time- consuming policy in a dynamic environment Decision, action Input variables on how current situation is captured Locally observed and communicated information An operational policy can be captured as a transformation from input to output.
18. Visualization in Industrial 4.0 Simulation as a virtual test bed 18 System state: s0 System state: s2 System state: s3 System state: s5 System state: s1 Revealed uncertainty (roll a dice) Operational decision (made by the policy) System state changes according to revealed uncertainty and operational decision made by the policy.
19. Visualization in Industrial 4.0 Inventory control example 19 Stock level & back orders Stock level & back orders Stock level & back orders Stock level & back orders Stock level & back orders How many items are demanded in this period How many to order at this period System state includes the stock level and the list of back orders, and changes according to revealed demand quantity and ordering decision made by the policy.
20. Visualization in Industrial 4.0 Simulation of random ordering policy 20 Ordering cost: 1.0 Holding cost: 0.025 Shortage penalty: 0.15 Lead time: 3 Demand: N(4, 4)
21. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 21
22. Visualization in Industrial 4.0 A classification of machine learning 22 Reinforcement learning Supervised learning Unsupervised learning Machine learning approaches Artificial intelligence techniques
23. Visualization in Industrial 4.0 Basic model States = {% , ' , … } Actions = {% , ' , … } Policy - = (|) Transition 23 = (3|, ) Reward (or cost) = (, ′) (or = (, ′) ) 23 st at policy st+1 transition reward at+1 policy st+2 transition reward at+2 policy
24. Visualization in Industrial 4.0 Value function and Q-learning Discounted sum of rewards to be obtained from now on (:= value!) ; = ; [>?% + B >?' + ' B >?C + ⋯ |> = ] = ; [>?% + B ; (>?% )|> = ] State-action value function (Q-table/Q-function) , = ; [>?% + B >?' + ' B >?C + ⋯ |> = , > = ] = ; [>?% + B max -3 (>?% , ′) |> = , > = ] Q-learning Refine approximated Q values step by step through simulation by: > , > ← (1 − ) B > , > + B [>?% + B max- 3 >?% , 3 ] 24
25. Visualization in Industrial 4.0 From Q-table to Q-function and DQN Q-table In each state, take the action which gives the maximum Q value in the corresponding row. When state is parameterized as = (% , ' , … ), Q-table can be generalized to a Q-function , = (, ). The famous DQN uses a deep neural network (a deep leaning model) for approximating this function. 25 a1 a2 a3 … s1 s2 s3 s4 s5 …
26. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 26
27. Visualization in Industrial 4.0 Inventory control example States Total number of items including back orders Actions How many to order? Transition Dependent on uncertain demand from downstream Cost (to be minimized) Ordering cost, holding cost, shortage penalty 27 15 20 policy 30 transition cost 0 policy 23 transition cost 0 policy
28. Visualization in Industrial 4.0 How Q-table is updated through learning 28
29. Visualization in Industrial 4.0 Cost reduction through learning 29 Ordering cost: 1.0 Holding cost: 0.025 Shortage penalty: 0.15 Lead time: 3 Demand: N(4, 4)
30. Visualization in Industrial 4.0 Simulation of a policy learned by RL 30 Ordering cost: 1.0 Holding cost: 0.025 Shortage penalty: 0.15 Lead time: 3 Demand: N(4, 4)
31. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 31
32. Visualization in Industrial 4.0 Serious games for skills analysis • Naive reinforcement learning becomes inefficient for obtaining a complex operational policy, especially in a multi-agent setting. • Skillful operational practices provide valuable data for streamlining such a tedious learning process. • That is, machine agents can learn form human specialists. • Serious games can be used as a tool for collecting virtual operational data efficiently. • They may also be used, the other way around, as a tool for training operational skills to novices. 32
33. Visualization in Industrial 4.0 Human learning and machine learning 33 Human skills Agent policy
34. Visualization in Industrial 4.0 Agenda • What are manufacturing systems? • What is to operate a manufacturing system? • How an operational policy can be modeled and tested? • What is reinforcement learning? • How to refine an operational policy by reinforcement learning? • How to learn from informal operational skills? • What did you learn today? 34
35. Visualization in Industrial 4.0 What did you learn today? Let‘s fill out by yourself! 35
36. Visualization in Industrial 4.0 36 Thank you! Questions & Comments are welcome