Enhancing Cyber Resilience with CIP Yoshitake Kobayashi, CIP TSC Chair Dinesh Kumar, CIP Security WG Lead September 3, 2024 

About speakers Dinesh Kumar CIP Security WG Lead Yoshitake Kobayashi CIP Technical Steering Committee Chair 2 

Agenda • Introduction to CIP • How CIP Enhances Cyber Resilience • CIP Security Working Group and IEC 62443 • Conclusion 3 

Platinum Members Silver Members 4 

Announcement • THURSDAY, 19 September • Time: 13:30 – 17:00 • Location: Austria Center Vienna Register here! CIP MINI SUMMIT Co-Located at Open Source Summit Europe 5 

Introduction to CIP and Cyber Resilience 

Our Civilization Runs on Linux: “Hidden” Industrial IoT Systems Rail automation Automatic ticket gates Vehicle control Transport Power Generation Turbine Control Energy Turbine Control Building automation Healthcare Broadcasting Others Industry automation Industrial communication CNC control Industry 7 

Civil Infrastructure an Increasing Target of Cybersecurity Threats Ref: 2024 Threat Report https://waterfall-security.com/2024-threat-report/ Cybersecurity Incidents in OT systems on public records since 2010 Cyberattacks are now nearly doubling annually. 8 

The Evolving Regulatory Landscape Cyber Resilience Act (CRA) and new Machinery Regulation (EU) 2023/1230 The President’s Executive Order (EO) 14028 on Improving the Nation’s Cybersecurity 9 

Understanding Cyber Resilience • The ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by cyber resources. (Ref: https://csrc.nist.gov/glossary/term/cyber_resiliency) • Key components • Prepare/Identify • Protect • Detect • Respond • Recover Prepare / Identify Protect Detect Respond Recover Cyber Resilience 10 

Establishing an Open Source Base Layer of industrial-grade software to enable the use and implementation of software building blocks for Civil Infrastructure Systems 11 

The key challenges • Apply IoT concepts to industrial systems • Ensure quality and longevity of products • Keep millions of connected systems secure • Product life-cycles of decades • Backwards compatibility • Standards • Reliability • Functional Safety • Real-time capabilities • Security & vulnerability management • Firmware updates • Minimize risk of regressions Sustainability Industrial gradeness Security 12 

CIP Core packages (tens) CIP kernel (10+ years maintenance, based on LTS kernels) Additional packages (hundreds) CIP Civil Infrastructure Platform Project (https://www.cip-project.org/) LTS Long Term Support CIP Open Source base layer company-specific middleware and applications Scope of a typical Linux distribution Layered Linux distribution for industrial products, utilizing and influencing the relevant Open Source projects: What is “Open Source Base Layer (OSBL)” ? 13 

OSS Open Source Software QA quality assurance SDK software development kit Corporate team/ central project Companies/ Divisions Business Units/ Products Firmware Update Security Hardening Container Runtime … Up to 70% effort reduction achievable for OSS license clearing and vulnerability monitoring, kernel and package maintenance, application adaptation and testing for an individual product. “distribution“ Kernel Base packages, SDK, Build chain, QA CIP Core packages (tens) Additional packages (hundreds) CIP Kernel (10+ years maintenance) Domain-specific extensions Domain-specific extensions … Mapping CIP into the company 14 

User space Kernel space Linux Kernel App container infrastructure (mid-term) App Framework (optionally, mid-term) Middleware/Libraries Monitoring Domain Specific communication (e.g. OPC UA) Shared config. & logging Real-time / safe virtualization Tools Concepts Tracing & reporting tools Configuration management Device management (update, download) Functional safety architecture/strategy, including compliance w/standards (e.g.,NERC CIP, IEC61508) Standardization collaborative effort with others License clearing Export Control Classification On-device software stack Product development and maintenance Application life-cycle management Multimedia Security Safe & Secure Update 6 2 5 Real-time support CIP Core Packages 3 1 Super Long Term Supported Kernel (STLS) 4 Test automation 3 Build environment (e.g. bitbake, dpkg) 1 3 Long-term support Strategy: security patch management Scope of activities 15 

Activities in CIP Project Workgroup Mission Industrial grade Sustain- ability Security Kernel Team • Providing CIP kernels with 10+ years maintenance period • Work with RT Linux project to upstream Real-time enhancement • Provide CIP SLTS kernel with real-time enhancement CIP Core • Provide a reference implementation with Debian based CIP core packages for testing CIP Testing • Providing a test environment to test the CIP kernel and CIP Core Security • Provide guidelines and reference implementations to help developers to meet cybersecurity standard requirements (IEC62443) SW update • Incorporate a common solution for software updates into CIP core 1 2 3 4 5 6 16 

How CIP enhances Cyber Resilience (1/2) • Long-term support and security updates • 10+ year maintenance period • Open source and upstream first principles • Community-driven improvements • Collaborative patching with upstream community • Faster vulnerability identification • Standardization and interoperability • OSBL as a common software platform • Reduced compatibility issues by CIP testing 17 

How CIP enhances Cyber Resilience (2/2) • Comprehensive Security Integration • Alignment with IEC 62443 standards • Security measures throughout system lifecycle • Threat modeling and risk assessment • Ongoing security validation and improvement • Continuous monitoring and adaptation • CVE monitoring for CIP kernel and CIP Core • Secure Software update mechanisms 18 

CIP Security Workgroup and IEC62443 

Building blocks of Cyber Resilience at component level 20 Identify Protect Detect Respond Recover 

IEC62443 Practices for Cyber Resiliency 21 Secure Implementation Security verification & Validation Security by design Security Management Management of Security related issues Security Update management 

CIP Cyber Resilience Support Identify & Protect IEC62443 practices/requirements CIP Adherence Human User identification and authentication • User identification by reusing Debian packages passwd, login, usermod, adduser • Provision of MFA • Support for latest Security packages openssl etc. Software process and Device authentication Identifier management, account management Use of Strong Cryptography, public key infrastructure 22 

CIP Cyber Resilience Support Detect, Respond & Recover 23 23 IEC62443 practices/requirements CIP Adherence Threat modelling, Risk assessment of third-party components Threat modelling for current processes and controls & mitigation Development environment security, File integrity Primarily gitlab based controls Auditable events, concurrent session control auditd package support CVE scanning & providing regular fixes CIP Kernel WG tracks CVE regularly and share fixes Continuous monitoring & Audit logs systemd-journald, main service for logs monitoring Control system backup & Restore There was no real use, hence CIP users to address it 

CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 24 CIP IEC62443-4-1 Final assessment status 24 Following IEC62443-4-1 processes were not feasible in CIP • Custom developed components from third party • Secure Design best practices • Defense in depth design in deployment • Penetration testing • Secure disposal guidelines 

CIP IEC62443-4-1 assessment recently concluded Most of the secure development practices can be met by reusing upstream as well as CIP development practices 25 

Good coverage of tests adding to Cyber resiliency • CIP Security Working Group investigated to find how many packages have tests available in Debian CI and package upstream • It seems a good percentage more than 85% packages have tests in either upstream or Debian CI • CIP members plan to work with upstream developers to enhance test coverage in future 26 CIP Security image package tests 

Ongoing effort for CIP IEC62443-4-2 27 Final assessment results IEC62443-4-2 final assessment SVV testing (in-progress) 

Further challenges for OSS community 28 • During CIP IEC assessment we learned few areas where OSS community must work further to strengthen security posture of OSS components • Review and rework communication is over mailing list, hence difficult to comply meeting the requirement “Track to closure security issue” • Lack of availability of common “Secure Coding Guidelines” • No central place to track if any Secure Design Principles are followed 

Cyber Resilience & Software Update 29 Vulnerability Management Regular software updates are crucial for patching known vulnerabilities. Rapid Response to Threats: The ability to quickly deploy updates in response to newly discovered vulnerabilities Adaptability: Regular updates allow systems to adapt to evolving threat landscapes, which is a core principle of cyber resilience. Incident Recovery: In the event of a successful attack, having the latest software updates can be crucial. Rollback Capability: A robust update system should include the ability to rollback changes if issues arise 

CIP supports Software Update 30 CIP Software update features • Basic Software updates provided by SWUpdate • Software update using A/B partition • Signed and encrypted image support TUF integration with CIP SWUpdate (WIP) • Hardening update delivery system. • Uses quorum of keys to sign artifacts, reducing the impact of key compromises. • Rotation the signing keys. WFX integration with TUF+SWUpdate in CIP (WIP) • Automate update workflow for fleet of devices at scale. • Manage update status to track any failed updates on the field. 

CIP IEC62443-4-x document management • Several requirements for maintaining IEC assessment documents • Maintain version of each document • Restricted access of some documents such as secure design and IEC information documents • Versions could be compared • Considering above aspects CIP has decided to maintain assessment documents • Most of the documents are created using Markdown to meet above requirements • CIP plans to migrate to readthedocs format in future • All documents maintained in CIP Gitlab repositories 31 

CVE handling in CIP • It’s also one of the IEC62443 requirement CIP intends to provide regular CVE updates on regular basis • https://gitlab.com/cip-project/cip-kernel/cip- kernel-sec CIP Kernel team does publish CVE updates via CIP developer ML • https ://gitlab.com/cip-project/cip-core/debian- cve-checker CIP Core members are working to provide tooling for CVE updates 32 

Reusing OSS components in IEC62443 layer • CIP developers do not modify any OSS component instead REUSES them • CIP users are recommended to use security configuration to implement product specific security policies • Follow upstream first policy and report issues in upstream for any changes • Primarily Debian packages are used to meet all IEC 62443 requirements • CIP users can further customize IEC layer by adding additional packages or even custom developed components 33 

CIP users benefit from CIP IEC62443 compliance 34 CIP being IEC62443 compliant will provide a secure foundation attested by IEC security capabilities Significantly reduces end product IEC62443 certification cost Rich set of documentation developed by CIP SWG and 

Advantages comparison CIP vs Non-CIP distributions Items CIP Non-CIP Dedicated kernel maintainers for SLTS up to 10 years × IEC62443-4-x assessed platform × Close monitoring of CVEs at user and kernel level × Extended support from Debian ELTS for specific packages × Regular automated testing on multiple SOCs with published test results on KernelCI × Strong support from big players of embedded system industry × 35 

Conclusion • CIP helps to enhance Cyber Resilience for your products and services • IEC62443-4-x compliant platform with Long-term support • Constantly striving to incorporate latest security features & updates • Engagement with security focused multiple Open-Source projects ● Open source and upstream first principles Collaboration is the key to ensure Cyber Resilience 36 

Join Now Join Now 37 Join your industry peers in helping build and shape the ecosystem for industrial grade software, its use cases and applications. Unite with other global leaders in power generation, oil and gas, communications and many other industries to establish the software building blocks for civil infrastructure. 

Meet us at CIP Mini Summit! • THURSDAY, 19 September • Time: 13:30 – 17:00 • Location: Austria Center Vienna Register here! CIP MINI SUMMIT Co-Located at Open Source Summit Europe 38 

To get the latest information, please contact: Other resources • CIP Mailing list: [email protected] • X: @cip_project • CIP web site: https://www.cip-project.org • CIP wiki: https://wiki.linuxfoundation.org/civilinfrastructureplatform/ • CIP source code - CIP GitLab: https://gitlab.com/cip-project - CIP kernel: https://git.kernel.org/pub/scm/linux/kernel/git/cip/linux-cip.git Contact Information and Resources 39 

Questions? 40 

Thank you! 41 

42