Civil Infrastructure Platform Industrial-Grade Linux Yoshitake Kobayashi, CIP TSC Chair November 2024 

Platinum Members Silver Members 2 Gold Members 

Introduction to CIP 3 

IoT today – Connecting Systems ● Multimodal transportation ● Intelligent traffic control ● Smart energy management ● Collect data to improve processes (cost, quality, speed) ● Minimize downtimes by predictive maintenance ● Find and rent cars via smartphone ● Monitor fleets and provide service Industry Smart City Connected Cars 

Our Civilization Runs on Linux®: “Hidden” Industrial IoT Systems Rail automation Automatic ticket gates Vehicle control Transport Power Generation Turbine Control Energy Turbine Control Building automation Healthcare Broadcasting Others Industry automation Industrial communication CNC control Industry 5 Linux is a registered trademark of Linus Torvalds. 5 

Civil Infrastructure has unique problems to solve: 

Kernel usage for products and its lifecycle 7 Year of the kernel release 

Civil Infrastructure an Increasing Target of Cybersecurity Threats Ref: 2024 Threat Report https://waterfall-security.com/2024-threat-report/ Cybersecurity Incidents in OT systems on public records since 2010 Cyberattacks are now nearly doubling annually. 8 Ref: https://edition.cnn.com/2021/05/08/politics/colonial-pipeline-cybersecurity-attack/index.html 

The Evolving Regulatory Landscape Cyber Resilience Act (CRA) The President’s Executive Order on Improving the Nation’s Cybersecurity 9 

The key challenges • Apply IoT concepts to industrial systems • Ensure quality and longevity of products • Keep millions of connected systems secure • Product life-cycles of decades • Backwards compatibility • Standards • Reliability • Functional Safety • Real-time capabilities • Security & vulnerability management • Firmware updates • Minimize risk of regressions Sustainability Industrial gradeness Security 10 

Establishing an Open Source Base Layer of industrial-grade software to enable the use and implementation of software building blocks for Civil Infrastructure Systems 11 

CIP Core packages (tens) CIP kernel (10+ years maintenance, based on LTS kernels) Additional packages (hundreds) CIP Civil Infrastructure Platform Project (https://www.cip-project.org/) LTS Long Term Support CIP Open Source base layer company-specific middleware and applications Scope of a typical Linux distribution Layered Linux distribution for industrial products, utilizing and influencing the relevant Open Source projects: What is “Open Source Base Layer (OSBL)” ? 12 

OSS Open Source Software QA quality assurance SDK software development kit Corporate team/ central project Companies/ Divisions Business Units/ Products Firmware Update Security Hardening Container Runtime … Up to 70% effort reduction achievable for OSS license clearing and vulnerability monitoring, kernel and package maintenance, application adaptation and testing for an individual product. “distribution“ Kernel Base packages, SDK, Build chain, QA CIP Core packages (tens) Additional packages (hundreds) CIP Kernel (10+ years maintenance) Domain-specific extensions Domain-specific extensions … Mapping CIP into the company 13 

User space Kernel space Linux Kernel App container infrastructure (mid-term) App Framework (optionally, mid-term) Middleware/Libraries Monitoring Domain Specific communication (e.g. OPC UA) Shared config. & logging Real-time / safe virtualization Tools Concepts Tracing & reporting tools Configuration management Device management (update, download) Functional safety architecture/strategy, including compliance w/standards (e.g.,NERC CIP, IEC61508) Standardization collaborative effort with others License clearing Export Control Classification On-device software stack Product development and maintenance Application life-cycle management Multimedia Security Safe & Secure Update 6 2 5 Real-time support CIP Core Packages 3 1 Super Long Term Supported Kernel (STLS) 4 Test automation 3 Build environment (e.g. bitbake, dpkg) 1 3 Long-term support Strategy: security patch management Scope of activities 14 

CIP Today focuses on Workgroup Mission/Activity Industrial grade Sustain- ability Security Kernel Team • Providing CIP kernels with 10+ years maintenance period • Work with RT Linux project to upstream Real-time enhancement • Provide CIP SLTS kernel with real-time enhancement CIP Core • Provide a reference implementation with Debian based CIP core packages for testing CIP Testing • Providing a test environment to test the CIP kernel and CIP Core Security • Provide guidelines and reference implementations to help developers to meet cybersecurity standard requirements (IEC62443) SW update • Incorporate a common solution for software updates into CIP core 1 2 3 4 5 6 15 

The backbone of CIP are the member companies Developers, Maintainers € ¥ $ £ Budget Open Source Projects (Upstream work) CIP Core packages CIP kernel Funding of selected projects Contribution & usage/integration 16 

Upstream first is CIP’s principle Upstream Projects LTS mainline 1 Upstream first 2 Use the upstream code 3 Integrate CIP Open Source Base Layer (OSBL) Contribute, Collaborate and use by CIP meta-debian SWUpdate 17 

Benefits to using CIP OSBL • Dedicated kernel maintainers for SLTS up to 10+ years • IEC62443-4-x assessed platform • Close monitoring of CVEs at user and kernel level • Extended support from Debian ELTS for specific packages • Regular automated testing on multiple SOCs with published test results at KernelCI • Strong support from big players of embedded system industry 18 

Solving the Key Challenges Together 

Why do companies engage? Share costs ● Super long term maintenance (10+ years) Establish and strengthen standards ● Security (IEC-62443-4-2), SW Update Boost relevant upstream projects ● Currently Debian, Real-time Linux, KernelCI, Reproducible Builds Improve quality ● Test infrastructure addressing industrial needs To build and shape the ecosystem for industrial grade software, its use cases, and applications. 

CIP Leadership Urs Gleim, Distinguished Engineer Connectivity and Edge Computing at Siemens AG CIP Governing Board Chair Yoshitake Kobayashi, General Manager at Toshiba Corporation CIP Technical Steering Committee Chair 

Case Study: Leveraging CIP ”CIP has grown into the Security of Linux and a defacto standard used at Siemens in hundreds of products”. - Urs Gleim, Head of Smart Embedded Systems at Siemens Summary of case study: Today, we have lots of products that are Linux-based. Many of these products were developed independently, which resulted in a huge number of different Linux versions and distributions. With all of these different versions in use, developers work to maintain all them in parallel. Furthermore, having more and more products connected increases the demand for providing security patches on short notice. Developers have worked in this way for a long time but this set-up is not scalable and cannot be managed like this in the future. To make matters more complicated, many companies are doing the same thing and maintaining several software stacks in parallel. The solution to this complicated issue is the harmonization of the Linux versions used in products. This is called the base layer and it’s the foundation of what CIP is working on. As member companies collaborate on this base layer will save money, resources and time in the long-run. 

Additional Member Spotlights: click logo 

Benefits for Semiconductors & Silicon Vendors Customers in industrial/infrastructure market expect Semiconductor vendors to provide a base software and support for their maintenance services due to its over 10 years product life cycle Maintenance Services: ● Software update services for bug/security fixes of the base software ● Application updates Mission critical products never accept kernel update, so longer term maintenance is required Leveraging CIP: ● Substantially reduces the additional development and maintenance burdens for customers associated with boosting the reliability and real-time responsiveness of software for industrial devices ● Enables manufacturers of industrial products to reduce the Total Cost of Ownership (TCO) ● Awareness in the industry market can be increased ● Expectation of Biz creation with the member companies ● Get reference information with regard to the requirements for next gen SoC 

CIP Membership Levels and Benefits 

Membership Benefits: Technical and Training Reference Boards: ● CIP supports a dedicated set of reference boards, i.e., all CIP projects are tested and released for these boards. Reference boards need to go through a nomination process, as defined by the TSC, before being accepted. ● Each Platinum Member can provide up to four different reference boards to be deployed and hosted for automated testing processes managed by the TSC; and ● each Gold Member can provide up to two different reference boards; ● Each member must provide their own physical reference boards. The member needs to provide four units of the same board for each type of board that is accepted. ● Boards which are reference boards already by end of December 2021 will stay reference boards, even if they had been nominated by Silver members or Associate members. If brought in by a Platinum Member they count as part of the total number of boards brought in by the respective member. ● Beyond that, additional reference boards, that are accepted through the nomination process, can be brought in by any member for an annual fee of $20,000 per board per year, for the time the board is supported. ● TSC also is in charge of a decommission process for situations in which a board needs to be removed. Training: ● CIP Platinum members entitled to 50-seat, annual subscription granting access to our library of eLearning courses and certification exams. Each seat gets access to up to two certification exams per year along with unlimited learning. (Benefit requires a signed MSA for training) 

Member Benefits: Events, Insights & Marketing Event Tickets ● A certain number of event tickets will be guaranteed to the members for events that CIP sponsors. The number of tickets is independent from the number of free tickets included in the respective sponsorship benefits, i.e. missing tickets will be purchased via CIP project budget. ● Platinum members always get 2 tickets minimum; and ● Gold members receive one ticket ● Remaining tickets, if available, will be distributed across Platinum, Gold, and Silver Members ● Members can donate their tickets to other persons, e.g., to Associate Members Leadership: ● Receive greater insight into CIP strategy and projects through engagement with the CIP leadership team. Platinum level members have the unique opportunity to customize their experience with CIP. The team will make themselves available to help achieve your strategic goals. CIP Team can help with guidance in open source contributions, new market creation, and/or open source project donation. Have ideas? Just ask! Marketing: ● Platinum members receive most prominent placement in displays of membership including website, landscape and marketing materials. CIP would like to emphasize those who contribute at the levels required to continue development and provide project stability in our marketing, PR, and event materials. ● Contribute 2 thought leadership articles on Linux.com (1M+ unique visitors per month) regarding how your organization is using CIP to accelerate innovation ● Platinum members are entitled to an individualized press release upon membership announcement with the PR team. All members are announced individually on the CIP website Blog ● All members have the opportunity to be featured in a Member Spotlight on the CIP website Blog 

Join Now Join Now 28 Join your industry peers in helping build and shape the ecosystem for industrial grade software, its use cases and applications. Unite with other global leaders in power generation, oil and gas, communications and many other industries to establish the software building blocks for civil infrastructure. 

To get the latest information, please contact: Other resources • CIP Mailing list: [email protected] • X: @cip_project • CIP web site: https://www.cip-project.org • CIP wiki: https://wiki.linuxfoundation.org/civilinfrastructureplatform/ • CIP source code - CIP GitLab: https://gitlab.com/cip-project - CIP kernel: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git Contact Information and Resources 29 

Questions? 30 30 

Thank you! 31 

CIP: Technical Deep Dive 

CIP SLTS kernel development (Upstream first development) Mainline / LTS CIP Kernel Team Maintainers and Developers 1 2 Providing CIP kernels with 10+ years maintenance period Kernel Team activities • Monitoring and Assessing Vulnerabilities • Continuously monitor security advisories. (e.g. CVEs) • Evaluate impact on CIP kernels. • Backporting Patches • Backport security fixes and important updates from mainline and LTS kernels to SLTS kernels. • Ensure compatibility and stability when integrating patches. • Upstream Contribution • Reviewing all patches to the latest linux-stable tree • Contributing features and fixes that need to be included in CIP kernels • Patch Management • Manage and track patches using repositories like cip-kernel-sec and patchwork • CIP SLTS Kernel release Patch review CVE Check Contributions Kernel Releases Branch / Platform Review and test results/ Fixes Feature mainlining Resources • CIP Kernel https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git • Configs, Tools and Review status https://gitlab.com/cip-project/cip-kernel 33 

CIP SLTS kernel development (Upstream first development) Items Achievements Kernel versions 4 versions with RT and without RT (v4.4, v4.19, v5.10, v6.1) Reference Platforms 10 boards (4 architectures: x86_64, ARMv7, ARMv8, RISCV64) Patch reviews Review patches for the CIP SLTS kernel versions CVE checking Approximately 600 fixes in 2024 on 6.1.y-cip Contributions 350+ for 6.1 Kernel Releases 54 (2022), 85 (2023), 50 (2024/09) Total: v4.4(89, rt49), v4.19(111,rt37), v5.10(50, rt37), 6.1(24, rt13) 1 3 4 2 5 1 1 2 3 4 2 5 1 34 

CIP SLTS kernel development (Upstream first development) Items Achievements Kernel versions 4 versions with RT and without RT (v4.4, v4.19, v5.10, v6.1) Reference Platforms 10 boards (4 architectures: x86_64, ARMv7, ARMv8, RISCV64) Patch reviews Review patches for the CIP SLTS kernel versions CVE checking Approximately 600 fixes in 2024 on 6.1.y-cip Contributions 350+ for 6.1 Kernel Releases 54 (2022), 85 (2023), 50 (2024/09) Total: v4.4(89, rt49), v4.19(111,rt37), v5.10(50, rt37), 6.1(24, rt13) 1 3 4 2 5 1 1 2 3 4 2 5 1 35 

CIP SLTS kernel development (Upstream first development) 1 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 LTS 4.4 4.19 5.10 6.1 CIP SLTS 4.4 4.19 5.10 6.1 We are here Upstream First Self-maintenance Upstream First Self-maintenance Upstream First Self-maintenance Maintained by the LTS Project Upstream First Self-maintenance Started self maintenance 1 2 36 

CIP SLTS kernel development (Upstream first development) Kernel versions and Projected EOL Version Maintainer(s) First Release Projected EOL Target Releases/Month SLTS v4.4 Nobuhiro Iwamatsu & Pavel Machek 2017-01-17 2027-01 1 SLTS v4.4-rt Pavel Machek 2017-11-16 2027-01 0.5 SLTS v4.19 Nobuhiro Iwamatsu & Pavel Machek 2019-01-11 2029-01 1 SLTS v4.19-rt Pavel Machek 2019-01-11 2029-01 0.5 SLTS v5.10 Nobuhiro Iwamatsu & Pavel Machek 2021-12-05 2031-01 1 SLTS v5.10-rt Pavel Machek 2021-12-08 2031-01 0.5 SLTS v6.1 Nobuhiro Iwamatsu & Pavel Machek 2023-07-14 2033-08 2 SLTS v6.1-rt Pavel Machek 2023-07-16 2033-08 1 37 1 1 2 

CIP Kernel Testing on Reference boards 4 1 Tested with standard Kernel configuration (non-RT) 2 Tested with Real-Time enabled Kernel configuration Supported Kernels Platform Architecture SLTS v4.4 SLTS v4.4-rt SLTS v4.19 SLTS v4.19-rt SLTS v5.10 SLTS v5.10-rt SLTS v6.1 SLTS v6.1-rt AM335x Beaglebone Black Armv7 Y Y1 Y Y1 Y T Cyclone V DE0-Nano-SoC Development Kit Armv7 N N Y Y1 Y T QEMU x86_64 Y Y1 Y Y1 Y T Y Y Armv7(a15) Y Y1 Y Y1 Y T Y Y Armv8(a53) Y Y1 Y Y1 Y T Y Y riscv64 N N N N Y N Y N RZ/G1M iWave Qseven Development Kit Armv7 Y Y2 Y Y2 Y Y RZ/G2M HopeRun HiHope Armv8 N N Y Y2 Y Y SIMATIC IPC227E x86-64 N N Y Y1 Y Y Y Y SIEMENS M-COM x86-64 N N N N Y Y Y Y OpenBlocks IoT VX2 x86-64 N N Y Y1 Y T Y T Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit Armv8 N N T T1 Y Y Candidate Reference Hardware Supported Kernels Platform Architecture SLTS v4.4 SLTS v4.4-rt SLTS v4.19 SLTS v4.19-rt SLTS v5.10 SLTS v5.10-rt Renesas RZ/Five EVK riscv64 N N N N Y T 1 2 38 

Unifying kernel configs 4 1 2 • Create a superset of all kernel configs par arch • Multiple reference board can be supported by one config • Status • Done for 86 config • Unification script merged as well • https://gitlab.com/cip-project/cip-kernel/cip-kernel- config/-/blob/master/README.merge_kconfig.md • Other arch need to work more to follow x86 39 Board 1 Board 2 Board 3 Board 1 Board 2 Board 3 config1 config2 config3 gen_conf 

CIP SLTS kernel development (Upstream first development) Items Achievements Kernel versions 4 versions with RT and without RT (v4.4, v4.19, v5.10, v6.1) Reference Platforms 10 boards (4 architectures: x86_64, ARMv7, ARMv8, RISCV64) Patch reviews Review patches for the CIP SLTS kernel versions CVE checking Approximately 600 fixes in 2024 on 6.1.y-cip Contributions 350+ for 6.1 Kernel Releases 54 (2022), 85 (2023), 50 (2024/09) Total: v4.4(89, RT 49), v4.19(111, RT 37), v5.10(50, RT 37), 6.1(24, RT 13) 1 3 4 2 5 1 3 4 2 5 1 40 4 1 2 

CIP SLTS kernel development (v4.4-cip) 1 2 • Averages • # of patches: 187.4 • # of CIP patches: 8.0 

CIP SLTS kernel development (v4.19-cip) 1 2 • Averages • # of patches: 190.3 • # of CIP patches: 4.2 

CIP SLTS kernel development (v5.10-cip) 1 2 • Averages • # of patches: 335.0 • # of CIP patches: 18.5 

CIP SLTS kernel development (v6.1-cip) 1 2 • Averages • # of patches: 481.9 • # of CIP patches: 15.9 

Weekly Trend of New Linux Kernel CVEs in 2024 45 Month 1st week 2nd week 3rd week 4th week 5th week Total Aaverage 2 7 15 130 152 50.7 3 277 5 49 60 391 97.8 4 137 82 108 10 337 84.3 5 207 32 6 685 149 1079 215.8 6 103 8 132 150 393 98.3 7 1 22 102 2 127 31.8 8 197 19 9 187 90 502 100.4 9 67 23 130 1 221 55.3 10 68 18 1 384 22 493 98.6 1 2 

Tools and Resources 4 1 2 • cip-kernel-sec • https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec • Tracks the status of security issues, identified by CVEID, in mainline, stable, and other configured branches. • kernel-cve-triage (Preview) • https://gitlab.com/cip-playground/kernel-cve-triage • Automated CVE Assessment • Evaluates CVEs against CIP kernel versions and configs. • Repository should become official CIP project soon 46 

CIP Core 3 Provide reference implementations with Debian-based CIP core packages for testing CIP Core WG activities • Implement and release the following reference images: • Generic profile ( isar-cip-core, which is actively under development) • Tiny profile ( deby, which is mostly in maintenance mode ) • Isar-cip-core refers Debian 8/10/11/12 • Monitoring and Assessing Vulnerabilities • Provide tooling for CVE updates https://gitlab.com/cip-project/cip-core/debian-cve-checker • Manage CIP Core packages • Add packages based on requests from WG and CIP members • Upstream Contribution • Work with Debian LTS/ELTS for long term maintenance • Using Reproducible build to ensure reproducibility 50 Build tool CIP Core Packages Reference Hardware Reference images CIP Core WG Testing (CI) Kernel Team SLTS Kernel CIP Testing Funding, contribution and collaboration Security/SW Update WG Requirements deploy Request to add packages 

• Funding Debian LTS and ELTS • Joined Debian LTS in 2018 (75 month) • Started participating Debian ELTS from Debian 8 • Focus on Debian 8 and Debian 10 • The requested package list is publicly available • https://gitlab.com/cip-project/cip-core/cip-pkglist • 84 packages • CIP Core images are now reproducible Ensuring sustainability through Collaboration 3 51 Artifact type Target machine Raw contents Filesystem Images Disk Images QEMU amd64 Reproducible Reproducible* Reproducible* QEMU arm64 Reproducible Reproducible* Reproducible* QEMU armhf Reproducible Reproducible* Reproducible* BBB Reproducible Reproducible* Reproducible* (*) All required patches had been already upstreamed ELTS Funding started Start Debian 10 ELTS Joined Debian LTS 

Software update working group 6 Incorporate a common solution for software updates into CIP core • E.g. Device management, Deployment, Safe update 52 CIP Software update features • Basic Software updates provided by SWUpdate • Software update using A/B partition • Signed and encrypted image support • Delta update supports TUF integration with CIP SWUpdate (WIP) • Hardening update delivery system. • Uses quorum of keys to sign artifacts, reducing the impact of key compromises. • Rotation the signing keys. WFX integration with TUF+SWUpdate in CIP (WIP) • Automate update workflow for fleet of devices at scale. • Manage update status to track any failed updates on the field. Reference H/W SWUpdate Secure boot Secure storage QEMU(*) Supported Supported Supported BBB Supported - - Renesas RZ/G2M Supported - - Siemens MCOM Supported Supported Supported Siemens IPC227E Supported - - 

Scope of Security working group Provide guidelines and reference implementations to help developers to meet cybersecurity standard requirements (IEC 62443) * this image represents the planning and is for illustrative purpose only Component User application User manual Design document Evaluation document User equipment S/W Document H/W Guideline and evidence Verified platform Compliant environment for evaluation Implement’n f. security Evaluation evidence Security requirements Application note Implement’n guideline Test cases Equipment for evaluation Reference implementations Application (sample) User manual Design document Evaluation document CIP Reference board Linux Kernel (CIP) Middleware / Libraries (CIP Core) CIP deliverables* 5 53 

IEC62443-4-1 Practices for Cyber Resiliency 54 Secure Implementation Security verification & Validation Security by design Security Management Management of Security related issues Security Update management 5 

55 55 CIP IEC62443-4-1 Final assessment status Following IEC62443-4-1 processes were not feasible in CIP • Custom developed components from third party • Secure Design best practices • Defense in depth design in deployment • Penetration testing • Secure disposal guidelines CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 

56 CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 

Package tests to meet IEC62443-4-2 57 5 Final assessment results IEC62443-4-2 final assessment SVV testing (in-progress) CIP Security image package tests • Investigated package tests availability in Debian CI and package upstream • More than 85% packages have tests • Total number of packages 142 • 19 packages need to be care • Plan to work with upstream developers to enhance test coverage 

CIP IEC62443-4-x document management • Several requirements for maintaining IEC assessment documents • Maintain version of each document • Restricted access of some documents such as secure design and IEC information documents • Versions could be compared • Considering above aspects CIP has decided to maintain assessment documents • Most of the documents are created using Markdown to meet above requirements • CIP plans to migrate to readthedocs format in future • All documents maintained in CIP Gitlab repositories 58 5 

Enhancing Cyber Resilience with CIP 59 

CIP enhances Cyber Resilience (1/2) • Long-term support and security updates • 10+ year maintenance period • Open source and upstream first principles • Community-driven improvements • Collaborative patching with upstream community • Faster vulnerability identification • Standardization and interoperability • OSBL as a common software platform • Reduced compatibility issues by CIP testing 60 

CIP enhances Cyber Resilience (2/2) • Comprehensive Security Integration • Alignment with IEC 62443 standards • Security measures throughout system lifecycle • Threat modeling and risk assessment • Ongoing security validation and improvement • Continuous monitoring and adaptation • CVE monitoring for CIP kernel and CIP Core • Secure Software update mechanisms 61 

Conclusion 62 • Our Civilization needs an Open Source Base Layer of industrial-grade software • Industrial-grade OSBL enhances sustainability and cyber resilience for your products and services • IEC62443-4-x compliant platform with Long-term support • Constantly striving to incorporate latest security features and updates • Engagement with multiple security focused open-source projects • CIP follows open source and upstream first principles Collaboration is the key to sustainable living 

Questions? 63 63 

Thank you! 64 

65