Civil Infrastructure Platform: Empowering Sustainable Living with Industrial Grade Linux Yoshitake Kobayashi, CIP TSC Chair (Toshiba) Open Source Summit Europe 2024, Vienna, Austria September 16, 2024 

About speaker Yoshitake Kobayashi CIP Technical Steering Committee Chair (Toshiba Corporation) 2 2 

Agenda • Introduction to CIP • Solving key challenges together • Enhancing Cyber Resilience with CIP • Conclusion 3 3 

Platinum Members Silver Members 4 

Announcement • THURSDAY, 19 September • Time: 13:30 – 17:00 • Location: Room 0.95/0.96 Austria Center Vienna 5 Register here! CIP MINI SUMMIT Co-Located at Open Source Summit Europe 5 

Introduction to CIP 6 

Our Civilization Runs on Linux®: “Hidden” Industrial IoT Systems Rail automation Automatic ticket gates Vehicle control Transport Power Generation Turbine Control Energy Turbine Control Building automation Healthcare Broadcasting Others Industry automation Industrial communication CNC control Industry 7 Linux is a registered trademark of Linus Torvalds. 7 

Kernel usage for products and its lifecycle 8 Year of the kernel release 

Civil Infrastructure an Increasing Target of Cybersecurity Threats Ref: 2024 Threat Report https://waterfall-security.com/2024-threat-report/ Cybersecurity Incidents in OT systems on public records since 2010 Cyberattacks are now nearly doubling annually. 9 

The Evolving Regulatory Landscape Cyber Resilience Act (CRA) The President’s Executive Order on Improving the Nation’s Cybersecurity 10 

Understanding Cyber Resilience • The ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by cyber resources. (Ref: https://csrc.nist.gov/glossary/term/cyber_resiliency) • Key components • Prepare/Identify • Protect • Detect • Respond • Recover Prepare / Identify Protect Detect Respond Recover Cyber Resilience 11 

The key challenges • Apply IoT concepts to industrial systems • Ensure quality and longevity of products • Keep millions of connected systems secure • Product life-cycles of decades • Backwards compatibility • Standards • Reliability • Functional Safety • Real-time capabilities • Security & vulnerability management • Firmware updates • Minimize risk of regressions Sustainability Industrial gradeness Security 12 

Establishing an Open Source Base Layer of industrial-grade software to enable the use and implementation of software building blocks for Civil Infrastructure Systems 13 

CIP Core packages (tens) CIP kernel (10+ years maintenance, based on LTS kernels) Additional packages (hundreds) CIP Civil Infrastructure Platform Project (https://www.cip-project.org/) LTS Long Term Support CIP Open Source base layer company-specific middleware and applications Scope of a typical Linux distribution Layered Linux distribution for industrial products, utilizing and influencing the relevant Open Source projects: What is “Open Source Base Layer (OSBL)” ? 14 

OSS Open Source Software QA quality assurance SDK software development kit Corporate team/ central project Companies/ Divisions Business Units/ Products Firmware Update Security Hardening Container Runtime … Up to 70% effort reduction achievable for OSS license clearing and vulnerability monitoring, kernel and package maintenance, application adaptation and testing for an individual product. “distribution“ Kernel Base packages, SDK, Build chain, QA CIP Core packages (tens) Additional packages (hundreds) CIP Kernel (10+ years maintenance) Domain-specific extensions Domain-specific extensions … Mapping CIP into the company 15 

User space Kernel space Linux Kernel App container infrastructure (mid-term) App Framework (optionally, mid-term) Middleware/Libraries Monitoring Domain Specific communication (e.g. OPC UA) Shared config. & logging Real-time / safe virtualization Tools Concepts Tracing & reporting tools Configuration management Device management (update, download) Functional safety architecture/strategy, including compliance w/standards (e.g.,NERC CIP, IEC61508) Standardization collaborative effort with others License clearing Export Control Classification On-device software stack Product development and maintenance Application life-cycle management Multimedia Security Safe & Secure Update 6 2 5 Real-time support CIP Core Packages 3 1 Super Long Term Supported Kernel (STLS) 4 Test automation 3 Build environment (e.g. bitbake, dpkg) 1 3 Long-term support Strategy: security patch management Scope of activities 16 

Activities in CIP Project Workgroup Mission Industrial grade Sustain- ability Security Kernel Team • Providing CIP kernels with 10+ years maintenance period • Work with RT Linux project to upstream Real-time enhancement • Provide CIP SLTS kernel with real-time enhancement CIP Core • Provide a reference implementation with Debian based CIP core packages for testing CIP Testing • Providing a test environment to test the CIP kernel and CIP Core Security • Provide guidelines and reference implementations to help developers to meet cybersecurity standard requirements (IEC62443) SW update • Incorporate a common solution for software updates into CIP core 1 2 3 4 5 6 17 

The backbone of CIP are the member companies Developers, Maintainers € ¥ $ £ Budget Open Source Projects (Upstream work) CIP Core packages CIP kernel Funding of selected projects Contribution & usage/integration 18 

Upstream first is CIP’s principle Upstream Projects LTS mainline 1 Upstream first 2 Use the upstream code 3 Integrate CIP Open Source Base Layer (OSBL) Contribute, Collaborate and use by CIP meta-debian SWUpdate 19 

Solving the Key Challenges Together 20 

CIP SLTS kernel development (Upstream first development) Mainline / LTS CIP Kernel Team Maintainers and Developers 1 2 Providing CIP kernels with 10+ years maintenance period Kernel Team activities • Monitoring and Assessing Vulnerabilities • Continuously monitor security advisories. (e.g. CVEs) • Evaluate impact on CIP kernels. • Backporting Patches • Backport security fixes and important updates from mainline and LTS kernels to SLTS kernels. • Ensure compatibility and stability when integrating patches. • Upstream Contribution • Contributing features and fixes that need to be included in CIP kernels • Patch Management • Manage and track patches using repositories like cip-kernel-sec and patchwork • CIP SLTS Kernel release Patch review CVE Check Contributions Kernel Releases Branch / Platform Review and test results/ Fixes Feature mainlining Resources • CIP Kernel https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git • Configs, Tools and Review status https://gitlab.com/cip-project/cip-kernel 21 

CIP SLTS kernel development (Upstream first development) Items Achievements Kernel versions 4 versions with RT and without RT (v4.4, v4.19, v5.10, v6.1) Reference Platforms 10 boards (4 architectures: x86_64, ARMv7, ARMv8, RISCV64) Patch reviews Review patches for the CIP SLTS kernel versions CVE checking Approximately 600 fixes in 2024 on 6.1.y-cip Contributions 350+ for 6.1 Kernel Releases 54 (2022), 85 (2023), 50 (2024/09) Total: v4.4(89, rt49), v4.19(111,rt37), v5.10(50, rt37), 6.1(24, rt13) 1 3 4 2 5 1 1 2 3 4 2 5 1 22 

CIP SLTS kernel development (Upstream first development) 1 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 LTS 4.4 4.19 5.10 6.1 CIP SLTS 4.4 4.19 5.10 6.1 We are here Upstream First Self-maintenance Upstream First Self-maintenance Upstream First Self-maintenance Maintained by the LTS Project Upstream First Self-maintenance Started self maintenance 1 2 23 

CIP SLTS kernel development (Upstream first development) Kernel versions and Projected EOL Version Maintainer(s) First Release Projected EOL Target Releases/Month SLTS v4.4 Nobuhiro Iwamatsu & Pavel Machek 2017-01-17 2027-01 1 SLTS v4.4-rt Pavel Machek 2017-11-16 2027-01 0.5 SLTS v4.19 Nobuhiro Iwamatsu & Pavel Machek 2019-01-11 2029-01 1 SLTS v4.19-rt Pavel Machek 2019-01-11 2029-01 0.5 SLTS v5.10 Nobuhiro Iwamatsu & Pavel Machek 2021-12-05 2031-01 1 SLTS v5.10-rt Pavel Machek 2021-12-08 2031-01 0.5 SLTS v6.1 Nobuhiro Iwamatsu & Pavel Machek 2023-07-14 2033-08 2 SLTS v6.1-rt Pavel Machek 2023-07-16 2033-08 1 24 1 1 2 

CIP Kernel Testing on Reference boards 4 1 Tested with standard Kernel configuration (non-RT) 2 Tested with Real-Time enabled Kernel configuration Supported Kernels Platform Architecture SLTS v4.4 SLTS v4.4-rt SLTS v4.19 SLTS v4.19-rt SLTS v5.10 SLTS v5.10-rt SLTS v6.1 SLTS v6.1-rt AM335x Beaglebone Black Armv7 Y Y1 Y Y1 Y T Cyclone V DE0-Nano-SoC Development Kit Armv7 N N Y Y1 Y T QEMU x86_64 Y Y1 Y Y1 Y T Y Y Armv7(a15) Y Y1 Y Y1 Y T Y Y Armv8(a53) Y Y1 Y Y1 Y T Y Y riscv64 N N N N Y N Y N RZ/G1M iWave Qseven Development Kit Armv7 Y Y2 Y Y2 Y Y RZ/G2M HopeRun HiHope Armv8 N N Y Y2 Y Y SIMATIC IPC227E x86-64 N N Y Y1 Y Y Y Y SIEMENS M-COM x86-64 N N N N Y Y Y Y OpenBlocks IoT VX2 x86-64 N N Y Y1 Y T Y T Zynq UltraScale+ MPSoC ZCU102 Evaluation Kit Armv8 N N T T1 Y Y Candidate Reference Hardware Supported Kernels Platform Architecture SLTS v4.4 SLTS v4.4-rt SLTS v4.19 SLTS v4.19-rt SLTS v5.10 SLTS v5.10-rt Renesas RZ/Five EVK riscv64 N N N N Y T 1 2 25 

Unifying kernel configs 4 1 2 • Create a superset of all kernel configs par arch • Multiple reference board can be supported by one config • Status • Done for 86 config • Unification script merged as well • https://gitlab.com/cip-project/cip-kernel/cip-kernel- config/-/blob/master/README.merge_kconfig.md • Other arch need to work more to follow x86 26 Board 1 Board 2 Board 3 Board 1 Board 2 Board 3 config1 config2 config3 gen_conf 

CIP SLTS kernel development (Upstream first development) Items Achievements Kernel versions 4 versions with RT and without RT (v4.4, v4.19, v5.10, v6.1) Reference Platforms 10 boards (4 architectures: x86_64, ARMv7, ARMv8, RISCV64) Patch reviews Review patches for the CIP SLTS kernel versions CVE checking Approximately 600 fixes in 2024 on 6.1.y-cip Contributions 350+ for 6.1 Kernel Releases 54 (2022), 85 (2023), 50 (2024/09) Total: v4.4(89, RT 49), v4.19(111, RT 37), v5.10(50, RT 37), 6.1(24, RT 13) 1 3 4 2 5 1 3 4 2 5 1 27 4 1 2 

Tools and Resources 4 1 2 • cip-kernel-sec • https://gitlab.com/cip-project/cip-kernel/cip-kernel-sec • Tracks the status of security issues, identified by CVEID, in mainline, stable, and other configured branches. • kernel-cve-triage (Preview) • https://gitlab.com/cip-playground/kernel-cve-triage • Automated CVE Assessment • Evaluates CVEs against CIP kernel versions and configs. • Repository should become official CIP project soon 28 

29 Tools and Resources (cip-kernel-sec) 4 1 2 ./scripts/report_affected.py v5.10.218-cip49 

30 Tools and Resources (kernel-cve-triage) 4 1 2 ./cve_triage.py -v triage --cip -s /home/yoshi/CIP/kernel:v5.10.218-cip49 --config /home/yoshi/CIP/cip-kernel-config/5.10.y-cip/x86/siemens_server_defconfig /home/yoshi/CIP/cip-kernel-sec/issues/CVE-2024-* 

31 Tools and Resources (kernel-cve-triage) 4 1 2 ./cve_triage.py analyze report.json-v5.10.218-cip49 

CIP Core 3 Provide reference implementations with Debian-based CIP core packages for testing CIP Core WG activities • Implement and release the following reference images: • Generic profile ( isar-cip-core, which is actively under development) • Tiny profile ( deby, which is mostly in maintenance mode ) • Isar-cip-core refers Debian 8/10/11/12 • Monitoring and Assessing Vulnerabilities • Provide tooling for CVE updates https://gitlab.com/cip-project/cip-core/debian-cve-checker • Manage CIP Core packages • Add packages based on requests from WG and CIP members • Upstream Contribution • Work with Debian LTS/ELTS for long term maintenance • Using Reproducible build to ensure reproducibility 32 Build tool CIP Core Packages Reference Hardware Reference images CIP Core WG Testing (CI) Kernel Team SLTS Kernel CIP Testing Funding, contribution and collaboration Security/SW Update WG Requirements deploy Request to add packages 

• Funding Debian LTS and ELTS • Joined Debian LTS in 2018 (75 month) • Started participating Debian ELTS from Debian 8 • Focus on Debian 8 and Debian 10 • The requested package list is publicly available • https://gitlab.com/cip-project/cip-core/cip-pkglist • 84 packages • CIP Core images are now reproducible Ensuring sustainability through Collaboration 3 33 Artifact type Target machine Raw contents Filesystem Images Disk Images QEMU amd64 Reproducible Reproducible* Reproducible* QEMU arm64 Reproducible Reproducible* Reproducible* QEMU armhf Reproducible Reproducible* Reproducible* BBB Reproducible Reproducible* Reproducible* (*) All required patches had been already upstreamed ELTS Funding started Start Debian 10 ELTS Joined Debian LTS 

Software update working group 6 Incorporate a common solution for software updates into CIP core • E.g. Device management, Deployment, Safe update 34 CIP Software update features • Basic Software updates provided by SWUpdate • Software update using A/B partition • Signed and encrypted image support • Delta update supports TUF integration with CIP SWUpdate (WIP) • Hardening update delivery system. • Uses quorum of keys to sign artifacts, reducing the impact of key compromises. • Rotation the signing keys. WFX integration with TUF+SWUpdate in CIP (WIP) • Automate update workflow for fleet of devices at scale. • Manage update status to track any failed updates on the field. Reference H/W SWUpdate Secure boot Secure storage QEMU(*) Supported Supported Supported BBB Supported - - Renesas RZ/G2M Supported - - Siemens MCOM Supported Supported Supported Siemens IPC227E Supported - - 

35 Demos at OSSEU CIP Booth 6 IoT use case Real time SW Update 

Scope of Security working group Provide guidelines and reference implementations to help developers to meet cybersecurity standard requirements (IEC 62443) * this image represents the planning and is for illustrative purpose only Component User application User manual Design document Evaluation document User equipment S/W Document H/W Guideline and evidence Verified platform Compliant environment for evaluation Implement’n f. security Evaluation evidence Security requirements Application note Implement’n guideline Test cases Equipment for evaluation Reference implementations Application (sample) User manual Design document Evaluation document CIP Reference board Linux Kernel (CIP) Middleware / Libraries (CIP Core) CIP deliverables* 5 36 

IEC62443-4-1 Practices for Cyber Resiliency 37 Secure Implementation Security verification & Validation Security by design Security Management Management of Security related issues Security Update management 5 

38 38 CIP IEC62443-4-1 Final assessment status Following IEC62443-4-1 processes were not feasible in CIP • Custom developed components from third party • Secure Design best practices • Defense in depth design in deployment • Penetration testing • Secure disposal guidelines CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 

39 CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 

Package tests to meet IEC62443-4-2 40 5 Final assessment results IEC62443-4-2 final assessment SVV testing (in-progress) CIP Security image package tests • Investigated package tests availability in Debian CI and package upstream • More than 85% packages have tests • Total number of packages 142 • 19 packages need to be care • Plan to work with upstream developers to enhance test coverage 

CIP IEC62443-4-x document management • Several requirements for maintaining IEC assessment documents • Maintain version of each document • Restricted access of some documents such as secure design and IEC information documents • Versions could be compared • Considering above aspects CIP has decided to maintain assessment documents • Most of the documents are created using Markdown to meet above requirements • CIP plans to migrate to readthedocs format in future • All documents maintained in CIP Gitlab repositories 41 5 

Enhancing Cyber Resilience with CIP 42 

CIP enhances Cyber Resilience (1/2) • Long-term support and security updates • 10+ year maintenance period • Open source and upstream first principles • Community-driven improvements • Collaborative patching with upstream community • Faster vulnerability identification • Standardization and interoperability • OSBL as a common software platform • Reduced compatibility issues by CIP testing 43 

CIP enhances Cyber Resilience (2/2) • Comprehensive Security Integration • Alignment with IEC 62443 standards • Security measures throughout system lifecycle • Threat modeling and risk assessment • Ongoing security validation and improvement • Continuous monitoring and adaptation • CVE monitoring for CIP kernel and CIP Core • Secure Software update mechanisms 44 

Conclusion 45 • Our Civilization needs an Open Source Base Layer of industrial-grade software • Industrial-grade OSBL enhances sustainability and cyber resilience for your products and services • IEC62443-4-x compliant platform with Long-term support • Constantly striving to incorporate latest security features and updates • Engagement with multiple security focused open-source projects • CIP follows open source and upstream first principles Collaboration is the key to sustainable living 

Join Now Join Now 46 Join your industry peers in helping build and shape the ecosystem for industrial grade software, its use cases and applications. Unite with other global leaders in power generation, oil and gas, communications and many other industries to establish the software building blocks for civil infrastructure. 

Meet us at CIP Mini Summit! • THURSDAY, 19 September • Time: 13:30 – 17:00 • Location: Austria Center Vienna 47 Register here! CIP MINI SUMMIT Co-Located at Open Source Summit Europe 

To get the latest information, please contact: Other resources • CIP Mailing list: [email protected] • X: @cip_project • CIP web site: https://www.cip-project.org • CIP wiki: https://wiki.linuxfoundation.org/civilinfrastructureplatform/ • CIP source code - CIP GitLab: https://gitlab.com/cip-project - CIP kernel: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git Contact Information and Resources 48 

Questions? 49 49 

Thank you! 50 

51