All day DevOps 2020 - CI in linux kernel development

Session Name: CI in Linux Kernel filesystem development with continuous performance analysis

"My plan is to cover the presentation based on the below four key points:
1. General discussion about continuous integration (CI) in kernel development:
- This discusses why CI is needed and how CIs are implemented in linux kernel development
2. Why performance is important and how often we test performance
- This section discusses the importantance and impacts of performance of kernel modules
3. How we process performance results with elastic (Open source version) and generate alerts (also open) based on performance improvement or degrade
- A simple explaination of how we use Elasticsearch for storing performance results and plotting performance results with Kibana.
Also discuss how alerts (Elastalerts) are implemented and notified via emails and in chat.
4. Future plans and improvements
- How Elastic's Machine learning (anomaly detection) can process performance results"

Transcript

1. TRACK: CI/CD CONTINUOUS EVERYTHING NOVEMBER 12, 2020 Anoop Vijayan @ Tuxera Oy CI in Linux kernel (filesystem) development 1
2. TRACK: CI/CD CONTINUOUS EVERYTHING • Linux kernel modules - CI • Performance analysis • Machine learning Topics 2
3. TRACK: CI/CD CONTINUOUS EVERYTHING • All Open source tools - DevOps stack • There is lots of interesting graphs at the end ;-) Viewers discretion advised 3
4. TRACK: CI/CD CONTINUOUS EVERYTHING • DevOps lead @ Tuxera Oy Finland • 15+ years of experience in developing and operating DevOps systems • Blogger, Open Source contributor during free time Tell me about yourself 4
5. TRACK: CI/CD CONTINUOUS EVERYTHING • Linux kernel modules - CI • Performance analysis • Machine learning Topics 5
6. TRACK: CI/CD CONTINUOUS EVERYTHING Is there a need for Continuous Integration (CI)? 6
7. TRACK: CI/CD CONTINUOUS EVERYTHING Who are we? 7
8. TRACK: CI/CD CONTINUOUS EVERYTHING File systems... 8 image: https://www.123rf.com/photo_87783152_phrase-what-is-that-in-black-ext-on-a-sticky-note-pinned-to-a-cork-notice-board.html
9. TRACK: CI/CD CONTINUOUS EVERYTHING What is a filesystem driver? 9 ref: https://en.wikipedia.org/wiki/Kernel_(operating_system) ref: Linux Network Architecture ISBN: 131777203 EAN: N/A Year: 2004
10. TRACK: CI/CD CONTINUOUS EVERYTHING What is a filesystem driver? 10 ref: https://en.wikipedia.org/wiki/Kernel_(operating_system) ref: Linux Network Architecture ISBN: 131777203 EAN: N/A Year: 2004
11. TRACK: CI/CD CONTINUOUS EVERYTHING What is the role of a filesystem driver 11 File System driver
12. TRACK: CI/CD CONTINUOUS EVERYTHING What are the filesystems available 12 Image: https://www.javatpoint.com/linux-file-system
13. TRACK: CI/CD CONTINUOUS EVERYTHING Where are we ? 13
14. TRACK: CI/CD CONTINUOUS EVERYTHING Embedded software life cycle 14 image: https://blog.nationwide.com/how-long-cars-last-infographic/
15. TRACK: CI/CD CONTINUOUS EVERYTHING Delivering File systems (kernel module) 15
16. TRACK: CI/CD CONTINUOUS EVERYTHING Deliverables 16 image: https://rescompany.com/products/resco-sms-ship-management-system/ Toolchain Kernel headers Software Kernel module Misc.
17. TRACK: CI/CD CONTINUOUS EVERYTHING Deliverables 17
18. TRACK: CI/CD CONTINUOUS EVERYTHING Deliverables 18 • Tool chains - ~2000 • Kernel headers - ~3000 • Processors - Intel, ARM, MIPS, PowerPC etc. • Architectures - 32-bit, 64-bit
19. TRACK: CI/CD CONTINUOUS EVERYTHING Deliverables vs Development 19
20. TRACK: CI/CD CONTINUOUS EVERYTHING CI in Linux kernel development 20 Toolchain Kernel headers Software Kernel module Misc. Dev/test FSTEST XFSTEST FAILSAFE
21. TRACK: CI/CD CONTINUOUS EVERYTHING Developments 21 • Tests - ~260 • Kernels - 10+3 • Hardwares - Virtual machines, real hardware • Architectures - 32-bit, 64-bit • Kernel debugging params - kasan, locks, etc.
22. TRACK: CI/CD CONTINUOUS EVERYTHING CI Testing 22
23. TRACK: CI/CD CONTINUOUS EVERYTHING CI pipeline 23 Deliverables check Development tests Static check Full test Result processing Archiving CI Server
24. TRACK: CI/CD CONTINUOUS EVERYTHING CI pipeline 24 Every commit Fast cycle Every night Slow cycle Deliverables check Development check Static check Full test Result processing Archiving CI Server
25. TRACK: CI/CD CONTINUOUS EVERYTHING • Linux kernel modules - CI • Performance analysis • Machine learning Topics 25
26. TRACK: CI/CD CONTINUOUS EVERYTHING • Somebody stole my data.. • My device is slow... Security and performance 26
27. TRACK: CI/CD CONTINUOUS EVERYTHING • Tuxera Filesystems supports all Linux Kernel Security Modules (LSM) • One of the most performant embedded filesystem in the market (more next slides..) Security and performance 27
28. TRACK: CI/CD CONTINUOUS EVERYTHING Performance measurement 28 Deliverables check Development tests Static check Full test Result processing Archiving CI Server
29. TRACK: CI/CD CONTINUOUS EVERYTHING Performance measurement 29
30. TRACK: CI/CD CONTINUOUS EVERYTHING Performance alerts 30
31. TRACK: CI/CD CONTINUOUS EVERYTHING • Find and fix performance bugs early in the development cycle. • Faster release cycle. • Motivates more automation and effective use of lab resources (Tuxera Oy) Benefits of performance testing in CI 31
32. TRACK: CI/CD CONTINUOUS EVERYTHING • Linux kernel modules – CI • Performance analysis • Machine learning Topics 32
33. TRACK: CI/CD CONTINUOUS EVERYTHING Machine learning – Anomaly detection 33
34. TRACK: CI/CD CONTINUOUS EVERYTHING Key take aways - 1 34 • Open source CI stack: ◦ Git – version control ◦ Jenkins – CI orchestrator ◦ Elastic stack – Reporting and alerting • Start with those on your own desktop with Docker containers ◦ Free and easily available
35. TRACK: CI/CD CONTINUOUS EVERYTHING • Modern CI can test any complex systems • Test fast, test early ◦ Performance testing in CI phase – shift left Key take aways - 2 35
36. TRACK: CI/CD CONTINUOUS EVERYTHING Reach me: https://twitter.com/maniankara https://fi.linkedin.com/in/anoopvijayan http://github.com/maniankara https://medium.com/@maniankara Q&A 36
37. TRACK: CI/CD CONTINUOUS EVERYTHING THANK YOU TO OUR SPONSORS 37