Moving Forward: Overcoming from Compatibility issues BoFs

Embedded Linux Conference 2011

Transcript

1. Moving Forward: Overcoming from Compatibility issues BoFs 13 Apr, 2011 Compatibility issues BoFs Yoshitake Kobayashi Advanced Software Technology Group Corporate Software Engineering Center TOSHIBA CORPORATION Copyright 2011, Toshiba Corporation.
2. Problem definition Libraries Application Test cases Product life cycle: more than 10 years 2 Embedded Linux Conference 2011 Hardware Kernel Libraries
3. Problem definition Libraries Application Test cases Libraries Application Test cases some years later Product life cycle: more than 10 years 3 Embedded Linux Conference 2011 Hardware Kernel Libraries Hardware Kernel Libraries Oh! No!
4. Problem definition Libraries Application Test cases Libraries Application Test cases some years later Product life cycle: more than 10 years 4 Embedded Linux Conference 2011 Hardware Kernel Libraries Hardware Kernel Libraries Oh! No! Maybe, I’m here
5. Problem definition Is new hardware introduction enough? No! Need to check supported hardware for the current kernel Mostly not…. Libraries Application Test cases To overcome the hardware discontinuation: 5 Embedded Linux Conference 2011 Newer Hardware Kernel Libraries Kernel Oh! No! Hardware Quite new Hardware or approx 5 years
6. Question 1: How to adapt the new hardware Which approach is the better? Upstream kernel driver backports on the old kernel Change the current product’s kernel to newer one 6 Embedded Linux Conference 2011
7. Question 1: How to adapt the new hardware Which approach is the better? Upstream kernel driver backports on the old kernel Change the current product’s kernel to newer one The answer is … I’m not sure 7 Embedded Linux Conference 2011 The correct answer depends on: Hardware specification User’s (or Programmer’s) requirement In this BoF, think about use newer kernel version to move forward
8. Question 2: Required tests What kind of test do I need to do to make sure the compatibility? 8 Embedded Linux Conference 2011
9. Question 2: Required tests What kind of test do I need to do to make sure the compatibility? 9 Embedded Linux Conference 2011 The following slides describe three aspects: API level Performance verification Service quality verification
10. Case study 1: API level tests Libraries Testcases Test environment Same libraries and testcases are used on each kernel version Libraries (lenny) Testcases’ Libraries (squeeze) Testcases’’ Libraries Testcases Libraries Testcases 10 Embedded Linux Conference 2011 Kernel (2.6.18-etch) Hardware Kernel (2.6.26-lenny) New Hardware Kernel (2.6.32-squeeze) New Hardware (lenny) (squeeze) C1 C2 C3
11. LTP resluts C1: 2.6.18-etch + Etch environment Error count that failed only on new hardware: 1 cron02 Note: This kernel doesn’t fully support new hardware C2: 2.6.26-lenny + Etch environment Error count that failed only on new hardware: 3 getcpu01, stime01, cron02 needs to run separately: mtest06, cron_deny01 11 Embedded Linux Conference 2011 needs to run separately: mtest06, cron_deny01 Note: This kernel supports almost all devices on new hardware C3: 2.6.32-squeeze + Ethc environment Error count that failed only on new hardware: 7 execve04, getcpu01, swapon03, sched_cli_serv, clock_gettime03, timer_create04 Note: This kernel supports almost all devices on new hardware
12. Determine the reason for the errors on 2.6.26 getcpu01 Only runs >2.6.20 Need NUMA support stime01 time() retuens stime()-1 A bug fix is available on 2.6.27.13 Easy to fix 12 Embedded Linux Conference 2011 Easy to fix
13. Case study 2: Performance verification CPU performance Has to be better than old one if the application’s CPU usage is high This is easy to fix Network throughput I/O throughput 13 Embedded Linux Conference 2011 The following aspects are important for real-time systems Scheduling latency Network latency …(any others?)
14. Latency test (cycle 300µs / cpu and memory load) CPU load 0 % 50 % 2.6.31.12 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 60000 70000 80000 90000 100000 2.6.31.12-RT 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 60000 70000 80000 90000 100000 counts counts counts counts latency [microsecs] latency [microsecs] 14 Embedded Linux Conference 2011 50 % 100 % 0 10000 20000 30000 40000 50000 60000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 counts counts counts counts latency [microsecs] latency [microsecs] latency [microsecs] latency [microsecs]
15. 60000 70000 80000 90000 100000 60000 70000 80000 90000 100000 Latency test(cycle 300µs / CPU load only) CPU負荷 0 % 50 % 2.6.31.12 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 2.6.31.12-RT 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 15 Embedded Linux Conference 2011 0 10000 20000 30000 40000 50000 60000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 200 400 600 800 1000 latency(μ秒) 回数 0 10000 20000 30000 40000 50000 60000 0 200 400 600 800 1000 latency(μ秒) 回数 50 % 100 %
16. Why this happens? Probably hardware problem Try to find the bottlenecks by ftrace This latency problem randomly happens in the kernel If same test runs on other machines, nothing happened In this case, just throw away the hardware or ask customer service 16 Embedded Linux Conference 2011 or ask customer service
17. Case study 3: Quality verification Quality verification for: File systems Long-term running 17 Embedded Linux Conference 2011
18. Results of data reliability tests 0.00 0.50 1.00 1.50 2.00 EXT3- ORDERED EXT3- JOURNAL * * * JFS XFS Error rate [%] 45.9% Error rate [%] 1.00 1.50 2.00 13.3% kernel 2.6.18 kernel 2.6.31 File size mismatch rate Data mismatch rate Point 1 ： ： ： ： An filesystem has different characteristics on different kernel Point 2: kernel version 18 Embedded Linux Conference 2011 Error rate [%] 0.00 0.50 1.00 1.50 2.00 EXT3- ORDERED EXT3- JOURNAL EXT4- JOURNAL EXT4- ORDERED EXT4- WRITEBACK * XFS BTRFS 82.4% 84.7% 43.4% 41.4% 43.2% Error rate [%] 0.00 0.50 EXT3- ORDERED EXT3- JOURNAL EXT4- JOURNAL EXT4- ORDERED * JFS XFS 2.6.31 kernel 2.6.33 Point 2: 2.6.33 has high error rate on ordered and writeback mode Point 3: Ext4-journal and Btrfs has good resluts File system types kernel version
19. Conclusion This BoF discussed what kind of tests are required to follow newer kernel version The following aspects are only examples API level compatibility Performance compatibility Service quality compatibility 19 Embedded Linux Conference 2011 Of course, application specific tests has to run
20. 20 Embedded Linux Conference 2011 2008 / 7 / 24 TOSHIBA Confidential