Using Embedded Linux for Infrastructure Systems Embedded Linux Conference Europe 2014 14 Oct 2014 Yoshitake Kobayashi 

2 Embedded Linux Conference Europe 2014 Scope of this presentation  Make a Place for collaboration  NOTE  This project currently belongs to CE workgroup, Linux Foundation  Anyone interested in this activities, please contact us 

3 Embedded Linux Conference Europe 2014 Outline  Definition of Social Infrastructure Systems  Problem statement  Expected goal  Project scope  Activities  Current status 

4 Embedded Linux Conference Europe 2014 Definition of Social Infrastructure Systems (SI systems)  Embedded systems which uses for the following infrastructure systems  Transport infrastructure  Energy infrastructure  Water management infrastructure  Communications infrastructure  Earth monitoring and measurement systems  The system includes  Automation systems  Controllers  PLC  Microcontrollers  Signal controller  Multi-purpose controller  Mini-server based controller (includes PC based controller)  Sensor network systems 

5 Embedded Linux Conference Europe 2014 An Example of Infrastructure System sensor Controller (e.g. PLC) 

6 Embedded Linux Conference Europe 2014 IoT? sensor Controller (e.g. PLC) THE INTERNET OF THINGS 

7 Embedded Linux Conference Europe 2014 Infrastructure System becomes a part of IoT sensor THE INTERNET OF THINGS THE INTERNET OF THINGS Moving Forward Controller (e.g. PLC) 

8 Embedded Linux Conference Europe 2014 Problem statement  The SI systems will be work with other systems such as:  Cloud computing  IoT / IoE related systems or devices 

9 Embedded Linux Conference Europe 2014 Problem statement  To connect with other systems, SI systems  Need to have same quality with new requirement  Need to have new technology to have new functionality  New requirements  Ensure a complicated performance requirement  Nondeterministic and deterministic programs runs on a same board  Collaborate with cloud platform  New functionality  Use fast non-volatile memory (e.g. MRAM, FeRAM) 

10 Embedded Linux Conference Europe 2014 Goal  Main goal  Develop reference implementations for SI system  Reference implementation may includes:  Linux kenel and filesystem for some target board  Target board: TBD  Open spec hardware is better for the implementation  Linux kernel related implementations should post to LKML for mainlining  Test-cases 

11 Embedded Linux Conference Europe 2014 Project scope  Security  Protect from irregular activities (e.g. viruses)  Integrity check  Dependability  Robustness  Run-Time Maintenance / Update  e.g. Kpatch / kGraft  Real-time  Hard real-time  Multi-purpose real-time system  Tests  Reliability test  Long Term Support  VLTS = Very Long Term Support  Legal/Compliance Issue  (Possibly) Functional Safety  The other scopes to be discussed 

12 Embedded Linux Conference Europe 2014 Activities  This project will be separated into several different sub-project by its key technology  Initial activity  Survey to determine current common key issues for SI systems  Create a mail list  F2F meeting at Linux Foudation’s conference  Embedded Linux Conference / LinuxCon  Workgroup meeting will be arranged  Reviewed by members of the CE Workgroup Architecture Group 

13 Embedded Linux Conference Europe 2014 Estimated output  Publicly available pages on the elinux wiki which includes the follows:  A list of issues  Specification for reference implementations  A list of link to reference implementations  All kernel related implementation should post to LKML  Reference implementations for key issues will be available on GitHub or alternative service 

14 Embedded Linux Conference Europe 2014 Current status  Under survey to determine actual issues by discussion  Start to call for participation 

15 Embedded Linux Conference Europe 2014 Project scope  Security  Protect from irregular activities (e.g. viruses)  Integrity check  Dependability  Robustness  Run-Time Maintenance / Update  e.g. kpatch / kGraft  Real-time  Hard real-time  Multi-purpose real-time system  Tests  Reliability test  Long Term Support  VLTS = Very Long Term Support  Legal/Compliance Issue  (Possibly) Functional Safety  The other scopes to be discussed 

16 Embedded Linux Conference Europe 2014 Project scope  Security  Protect from irregular activities (e.g. viruses)  Integrity check  Dependability  Robustness  Run-Time Maintenance / Update  e.g. kpatch / kGraft  Real-time  Hard real-time  Multi-purpose real-time system  Tests  Reliability test  Long Term Support  VLTS = Very Long Term Support  Legal/Compliance Issue  (Possibly) Functional Safety  The other scopes to be discussed 

17 Embedded Linux Conference Europe 2014  Real-time  Improve scheduring granularity for SCHED_DEADLINE  Hardware resource isolation to run RT and GP process  Test  Deta reliability evaluation for Linux filesystems  Other  Long-term testing by accelerated kernel 

18 Embedded Linux Conference Europe 2014 Issue on SCHED_DEADLINE Wakeup Period Task execution Time dl_timer 1.5ms task_tick_dl 1ms Wakeup dl_timer 1.5ms 1ms Step1: Refill budget Step2: Use budget Period 2ms  Difficult to keep task’s budget if the budget has micro seconds granularity Period: 1ms, Budget: 0.5ms 

19 Embedded Linux Conference Europe 2014 Support for micro seconds granularity Period: 1ms, Budget: 0.5ms Wakeup Period Task execution Set HRTICK here Step1: Refill budget Step2: Use budget Time 3.2ms 1ms 2ms 3ms Wakeup  When a task’s budget is less than 1ms, set HRTICK for the rest of budget Improvement of Scheduling Granularity for Deadline Scheduler http://elinux.org/images/9/9c/Improvement_of_Scheduling_Granularity_for_Deadline_Scheduler.pdf Source code is available on GitHub ( https://github.com/ystk/sched-deadline ) 

20 Embedded Linux Conference Europe 2014 Let’s play a movie Modified Original  Play the movie about 3 times faster than normal speed  Requirement for a frame decoding  Period: 10ms, budget: 3.6ms  When a frame decode is not finished in a period, the player shows “DEADLINE MISS” on screen and add 3ms penalty “Big Buck Bunny” http://www.bigbuckbunny.org/ (c) copyright 2008, Blender Foundation License: Creative Commons Attribution 3.0 

21 Embedded Linux Conference Europe 2014  Realtime  Improve scheduring granularity for SCHED_DEADLINE  Hardware resource isolation to run RT and GP process  Reliability  Deta reliability evaluation for Linux filesystems  Functionality  Long-term testing by accelerated kernel 

22 Embedded Linux Conference Europe 2014 RT process and GP process runs on a board CPU Device 1 Device 2 Hardware Applications OS Linux RT App Core 1 Core 2 GP App  Goal  Run RT processes and GP processes on a hardware platform CPU Device 1 Device 2 RT App RT core GP core GP App Linux Isolate from GP application’s behavior 

23 Embedded Linux Conference Europe 2014 An example for isolation demo Eth0 SD UART Core1 Core0 Web server RT control Pandaboard Workload (Web clients) Puppy Motor Sensors (Gyroscope, etc.) Motor torque Sensor values CAN RS232C Serial-CAN converter Point RT Control and UART interrupts are isolated from the Web server and other interrupts. IRQ IRQ 

24 Embedded Linux Conference Europe 2014 Demo (Not Isolated) 

25 Embedded Linux Conference Europe 2014 Demo (Isolated) 

26 Embedded Linux Conference Europe 2014 An issue while control the robot Latency [μs] Latency for RT app while running with GP appliation Evaluation Environment Hardware ・ Pandaboard - ARM Dual core 1.2GHz - Memory 1GB - SDcard OS ・ Linux 3.6.11.6-RT38 Application ・ RT App - Periodic task Period = 300 micro sec ・ GP App - Random I/O on SDcard 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 Number of samples Before (CPUset) This latency causes a failure 

27 Embedded Linux Conference Europe 2014 Realtime performance improvement with Timer Shielding Timers Timers Timer management Hardware Applications Linux Kernel RT core GP core RT App GP App Timers Timers Timer management Hardware Applications Linux Kernel RT core GP core RT App GP App Migrator Timer Shield ON Timer Shield OFF Hardware Partition Management （Create / Modify / Delete） Migrator Prohibit to resister non RT related timers Migrate Migrate non RT relate timers API to manage hardware partitions Register Issue Some kernel task does not able to allocate to target core An Essential Relationship between Real-time and Resource Partitioning http://www.elinux.org/images/6/6a/ELCE2013-Kobayashi-An_Essential_Relationship_between_Real-time_and_Resource_Partitioning.pdf 

28 Embedded Linux Conference Europe 2014 An issue while control the robot 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 Number of samples Latency [μs] After Latency is less than 60us Latency [μs] Latency for RT app while running with GP appliation Evaluation Environment Hardware ・ Pandaboard - ARM Dual core 1.2GHz - Memory 1GB - SDcard OS ・ Linux 3.6.11.6-RT38 Application ・ RT App - Periodic task Period = 300 micro sec ・ GP App - Random I/O on SDcard 0 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 600 700 800 900 Number of samples Before (CPUset) 

29 Embedded Linux Conference Europe 2014  Realtime  Improve scheduring granularity for SCHED_DEADLINE  Hardware resource isolation to run RT and GP process  Test  Deta reliability evaluation for Linux filesystems  Other  Long-term testing by accelerated kernel 

30 Embedded Linux Conference Europe 2014 Data reliability test Checker Target file LOG file AAAAA BBBBB CCCCC DDDDD EEEEE OK FFFFF AAAAA BBBBB CCCCC DDDDD EEEEE OK AAAAA BBBBB CCCCC DDDDD AAAAA NG AAAAA BBBBB CCCCC DDDDD NG ? size mismatch data mismatch Estimated file contents Estimated file size  Power off while running writer processes  Verify all data  File size  Contents Evaluation of Data Reliability on Linux File Systems http://elinux.org/images/2/26/Evaluation_of_Data_Reliability-ELC2010.pdf Test case is available on GitHub ( https://github.com/ystk/fs-test ) 

31 Results of data reliability tests on Linux filesystems 0.00 0.50 1.00 1.50 2.00 EXT3- ORDERED EXT3- JOURNAL * * * JFS XFS Error rate [%] 45.9% Error rate [%] 0.00 0.50 1.00 1.50 2.00 EXT3- ORDERED EXT3- JOURNAL EXT4- JOURNAL EXT4- ORDERED EXT4- WRITEBACK * XFS BTRFS 82.4% 84.7% 43.4% 41.4% 43.2% Error rate [%] 0.00 0.50 1.00 1.50 2.00 EXT3- ORDERED EXT3- JOURNAL EXT4- JOURNAL EXT4- ORDERED * JFS XFS 13.3% kernel 2.6.18 kernel 2.6.31 kernel 2.6.33 File size mismatch rate Data mismatch rate Point 1： An filesystem has different characteristics on different kernel Point 2: 2.6.33 has high error rate on ordered and writeback mode Point 3: Ext4-journal and Btrfs has good resluts File system types kernel version Ext3 (O) Ext3 (J) Ext4 (J) Ext4 (O) Ext4 (W) JFS XFS BTRFS NA NA NA 

32 Embedded Linux Conference Europe 2014  Realtime  Improve scheduring granularity for SCHED_DEADLINE  Hardware resource isolation to run RT and GP process  Test  Deta reliability evaluation for Linux filesystems  Other  Long-term testing by accelerated kernel 

33 Embedded Linux Conference Europe 2014 Long-term running test by accelerated kernel START GOAL Accelerate Detect clock counter related errors (e.g. time_t) Accelerate clock counter jiffies = jiffies + (1 * ratio) Issue • Long-term testing takes really long time → We want to have results as fast as possible Things that NOT able to be accelerated. • CPU frequency • HDD or SSD access speed • Network link speed • … 

34 Embedded Linux Conference Europe 2014 Evaluation with accelerated kernel Xdaliclock works as a stopwatch About 40 times faster to get 100% CPU usage returned incorrect value after about 450 days Linux Kernel Acceleration for Long-term Testing, ELC Europe 2010, Oct. 2010. http://elinux.org/images/6/6d/Linux_Kernel_Acceleration_for_Long-term_Testing.pdf 

35 Embedded Linux Conference Europe 2014 Enough?  Real-time latency is short enough  Data is safe enough  System works long enough 

36 Embedded Linux Conference Europe 2014 Project scope  Security  Protect from irregular activities (e.g. viruses)  Integrity check  Dependability  Robustness  Run-Time Maintenance / Update  e.g. kpatch / kGraft  Real-time  Hard real-time  Multi-purpose real-time system  Tests  Reliability test  Long Term Support  VLTS = Very Long Term Support  Legal/Compliance Issue  (Possibly) Functional Safety  The other scopes to be discussed 

37 Embedded Linux Conference Europe 2014 Summary  There are many issues need to be solved for SI system  CE workgroup is considering to have sub project for it  Current status  Under survey for actual issues  Start call for participation  Initial activity will be the follows:  Clarifying issues to be solved  Create a mail list  Have F2F meetings at Linux Foundaton’s conference  Estimated output  Publicly available pages on the elinux wiki or Linux Foundation’s web site which includes the follows:  A list of issues  Specification for reference implementations  A list of link to reference implementations  All kernel related implementation should post to LKML  All source code for key issues will be opened 

38 Embedded Linux Conference Europe 2014 Discussion  Any comments and suggestions are welcome  Contact information  To get the latest information, please send an email to the following adderss:  Yoshitake Kobayashi [email protected]  Noriaki Fukuyasu [email protected] 

39 Embedded Linux Conference Europe 2014