Applying serverless architecture pattern to distributed data processing

The talk was presented at PyCon APAC 2018. This talk is all about serverless and how you can apply "serverless architecture pattern" to the distributed data processing.

Transcript

1. Copyright © 2018, Oracle. All rights reserved. | 1 Applying serverless architecture pattern to distributed data processing Denis Makogon Principal member of technical stuff Oracle Corp. June 2, 2018
2. Copyright © 2018, Oracle. All rights reserved. | From multipurpose microservices to granular processing units 2
3. Copyright © 2018, Oracle. All rights reserved. | From monolith apps to microservices to what? 3
4. Copyright © 2018, Oracle. All rights reserved. | I know some people really hate serverless term since there are indeed servers running in the stack, it’s just that I don’t have to know they are there 4
5. Copyright © 2018, Oracle. All rights reserved. | “serverless architecture pattern” What is that? 5
6. Copyright © 2018, Oracle. All rights reserved. | 6
7. Copyright © 2018, Oracle. All rights reserved. | 7 Map-Reduce but serverless!
8. Copyright © 2018, Oracle. All rights reserved. | Tensor but scalable! 8
9. Copyright © 2018, Oracle. All rights reserved. | FIFO/FILO queues but distributed! 9
10. Copyright © 2018, Oracle. All rights reserved. | 10 Was made by developer for developers, because they care!
11. Copyright © 2018, Oracle. All rights reserved. | Fn project Open source Container-native Developer-oriented, developer-friendly Programming language-agnostic runtime 1 2 3 4 11
12. Copyright © 2018, Oracle. All rights reserved. | FDKs 12
13. Copyright © 2018, Oracle. All rights reserved. | 13 Stop talking. Do something!
14. Copyright © 2018, Oracle. All rights reserved. | Videobox app 14
15. Copyright © 2018, Oracle. All rights reserved. | Videobox from video to frames-per-second detect-detect-detect, put logo assemble 1 second assemble all “1-second”s clean-up 1 2 3 15 4 5
16. Copyright © 2018, Oracle. All rights reserved. | BOT
17. Copyright © 2018, Oracle. All rights reserved. | frame splitter bucket daemon object detect segment assembler segments assembler bucket cleaner
18. Copyright © 2018, Oracle. All rights reserved. | frame splitter object detect object detect object detect object detect object detect object detect object detect object detect segment assembler segment assembler segment assembler segment assembler segment assembler segment assembler segment assembler segment assembler segment assembler bucket daemon segments assembler bucket cleaner
19. Copyright © 2018, Oracle. All rights reserved. | 19 1 frame splitter 46 batch object detections 46 segment assemblers continuos backet daemon 1 bucket cleaner 1 segments assembler
20. Copyright © 2018, Oracle. All rights reserved. | 20 True power of serverless: do more with less!
21. Copyright © 2018, Oracle. All rights reserved. | Serverless data processing model Scale out on demand, scale in on idle Sequence or pipeline Stateless 1 2 3 21
22. Copyright © 2018, Oracle. All rights reserved. | From zero to serverless 22 fn init --runtime python3.6 first-function fn deploy --app first-app --registry `whoami` echo ‘{“name”: “Denis”}’ | fn call first-app first-function
23. Copyright © 2018, Oracle. All rights reserved. | From zero to serverless 23
24. Copyright © 2018, Oracle. All rights reserved. | 24 Screenshot of new fdk-python feature Native coroutine support
25. Copyright © 2018, Oracle. All rights reserved. | Something special for Python developers! 25 Screenshot of new fdk-python feature
26. Copyright © 2018, Oracle. All rights reserved. | 26 To pickle! Back and forth.
27. Copyright © 2018, Oracle. All rights reserved. | Serverless makes you to pick technologies more wisely! 27
28. Copyright © 2018, Oracle. All rights reserved. | VS
29. Copyright © 2018, Oracle. All rights reserved. | If you’re good at machine learning, data processing — share your knowledge by developing a portfolio with functions. 29
30. Copyright © 2018, Oracle. All rights reserved. | 30 Share knowledge and best practices within reproducible runtime.
31. Copyright © 2018, Oracle. All rights reserved. | Key takeaways Technology adoption no longer a concern. Granularity means better scaling. Function has no application limits. Delegate the workload, keep an application entry point thin. 1 2 3 31 4
32. Copyright © 2018, Oracle. All rights reserved. | Things that you better take a look at! https://medium.com/@denismakogon https://medium.com/@fnproject https://github.com/fnproject https://github.com/denismakogon/videobox https://www.katacoda.com/fnproject 1 2 3 32 4 5
33. Questions?