Securing and managing dependencies at scale

Transcript

1. Securing and managing
   dependencies at scale
   Ho YuehHsun
   

   View Slide

2. Ho YuehHsun
   - Joined PayPay at 2019/08.
   - DevSecOps team.
   - Focus on Dev side security.
   - Interests: Travel in Japan.
   

   View Slide

3. - Reduce 70% percentage of
   vulnerabilities.
   - Finish migration on ~180
   services in a quarter.
   - Finished all front-facing
   services.
   - Finished all backend services.
   Reduce vulnerabilities at scale
   Backend migration
   Front-end migration
   

   View Slide

4. ● A vulnerability is a bug which can be exploited by an attacker.
   ○ Known vulnerabilities is one of the most critical cyber risks.
   ○ CVEs (Common Vulnerabilities and Exposure) identify and catalog
   vulnerabilities.
   ■ Severity level: Critical, High, Medium, Low
   ○ An attack can facilitate data loss or server takeover.
   ■ Equifax breach in 2017
   ○ Known vulnerabilities have exploit kits.
   Vulnerabilities and Exploits
   

   View Slide

5. ● How to prevent:
   ○ Reducing threat surface.
   ■ Remove unneeded dependencies, components, files, and
   documentation.
   ○ Continuously monitor CVEs for vulnerabilities in
   components.
   ○ Fix or upgrade underlying platforms, frameworks, and
   dependencies regularly.
   Vulnerabilities and Exploits
   

   View Slide

6. - Micro-Service architecture
   - Services built from many
   3rd-party and open-source
   dependencies.
   PayPay’s infrastructure
   Payment
   Finnet Wallet
   App POS Online
   ©PayPay Corporation.All rights reserved.
   CLM
   ...
   

   View Slide

7. ● Upgrade the version in gradle build file
   ○ Teams doesn’t always know the best way to fix a vulnerability.
   ■ One by one, each by each
   ■ Resolution Strategy (Gradle)
   impl 'com.fasterxml.jackson.core:jackson-core:2.12.2' // previously 2.12.0
   How to upgrade to a non-vulnerable version
   configurations.all {
   resolutionStrategy.eachDependency {
   DependencyResolveDetails details ->
   if (details.requested.group == 'org.apache.tomcat.embed') {
   details.useVersion '8.5.60'
   details.because 'Security Fixes (Multiple Issues)'
   }
   }
   }
   

   View Slide

8. Dependency Management in build system
   ● Maven
   ○ BOM (Bills of Materials):
   ■ BOM is a kind of POM file that is used to suggest the versions of a
   projects dependencies and provide a central place to define and update
   those versions.
   ■ The version can also be overridden.
   

   View Slide

9. spring-boot-dependencies BOM:
   
   ...
   
   org.springframework.boot
   spring-boot-test
   2.3.11.RELEASE
   
   
   org.springframework.boot
   spring-boot-test-autoconfigure
   2.3.11.RELEASE
   
   
   org.springframework.boot
   spring-boot-actuator
   2.3.11.RELEASE
   
   
   org.springframework.boot
   spring-boot-actuator-autoconfigure
   2.3.11.RELEASE
   
   ...
   
   

   View Slide

10. ● Import a BOM in Maven
    ● Don’t need to declare the version if the BOM is imported
    
    
    org.springframework.boot
    spring-boot-starter
    
    
    org.springframework.boot
    spring-boot-starter-web
    
    
    
    
    
    
    org.springframework.boot
    spring-boot-dependencies
    2.3.11.RELEASE
    pom
    import
    
    
    
    

    View Slide

11. Dependency Management in build system
    - Gradle
    - Dependency Management Plugin
    - A plugin created by the Gradle team to implement Maven’s dependency
    management.
    plugins { id 'org.springframework.boot' version '2.3.11.RELEASE' }
    ext { springBootVersion = ‘2.3.11.RELEASE’ }
    dependencyManagement {
    imports {
    mavenBom('org.springframework.boot:spring-boot-dependencies:${springBootVersion}')
    }
    }
    

    View Slide

12. ● Gradle Platform
    ○ A platform is a software component in gradle system which can be used to
    control, suggest, enforce dependency versions.
    ○ Maven BOM is also a kind of platform that gradle supports.
    ext {
    springBootVersion = ‘2.3.11.RELEASE’
    }
    implementation platform(“org.springframework.boot:spring-boot-dependencies:${springBootVersion}”)
    

    View Slide

13. Solution: BOM
    ● A dependency management artifact that allows us to control the versions for a
    set of core dependencies.
    ● Advantages:
    ○ Reduce the burden for team to maintain dependencies individually.
    ○ Define non-vulnerable versions of dependencies.
    ○ Introduce consistency of dependencies across services.
    ○ Version can be overridden.
    

    View Slide

14. Solution: BOM
    impl ('jp.ne.paypay:paypay-lib1:2.3.5')
    - CVE1, CVE2
    impl ('jp.ne.paypay:paypay-lib2:4.1.7')
    - CVE3
    impl ('com.foo:foo-bar:1.0.1')
    impl platform('paypay-bom:1.0.0')
    impl ('jp.ne.paypay:paypay-lib1') (0 CVE)
    impl ('jp.ne.paypay:paypay-lib2') (0 CVE)
    impl ('com.foo:foo-bar')
    PayPay BOM
    - 'jp.ne.paypay:paypay-lib1:2.4.0'
    CVE1, CVE2
    - 'jp.ne.paypay:paypay-lib2:4.2.0'
    CVE3
    - 'com.foo:foo-bar:1.0.1'
    

    View Slide

15. Solution: BOM
    PayPay BOM
    lib1:1.0.0
    - Fix CVE1 CRITICAL
    lib2:1.1.0
    - Fix CVE2 CRITICAL
    lib3:2.1.1
    - Fix CVE3 CRITICAL
    …
    libX:2.0.0
    - Fix CVEX LOW
    Payment
    Finnet
    Wallet
    POS
    Online
    ©PayPay Corporation.All rights reserved.
    CLM
    ...
    

    View Slide

16. Create the BOM
    ● Preliminary work:
    ○ Build System: gradle or maven?
    ○ Dependency selection.
    ○ Dependency constraints selection.
    ○ Naming and versioning.
    

    View Slide

17. ○ Spring Boot and Spring Cloud dependencies based.
    ■ Spring Boot: 2.3.X, 2.4.X.
    ■ Spring Cloud: corresponding versions.
    ○ Internal PayPay Libraries.
    ○ Internal PayPay BOMs.
    ○ Fixed version of vulnerable dependencies that is critical or high
    severity level.
    Create the BOM: dependency selection
    

    View Slide

18. ● Dependency constraints selection
    ○ Vulnerable dependencies:
    ■ The minimum version constraints that could fix the
    vulnerability.
    ■ Newest version is preferred.
    com.lib:lib.bar:1.2.0
    - CVE 1
    - CVE 2
    com.lib:lib.bar:1.2.1
    - CVE 1
    - CVE 2
    com.lib:lib.bar:1.3.0
    - CVE 1
    - CVE 2
    ✅
    Create the BOM: dependency constraint selection
    

    View Slide

19. ○ Name: paypay-spring-bom
    ○ Version:
    ■ Format:
    ■ Use Spring-Boot version as part of the BOM version.
    ● Developers can easily determine the Spring-Boot version.
    ● example:
    ■ Add a patch number: PPX to the version indicating minor changes.
    ● X is a number and increased by 1 when there’s a minor change.
    ${SpringBootVersion}-PPX
    Create the BOM: naming and versioning
    paypay-spring-bom:2.3.6
    

    View Slide

20. BOM: 2.3.6-PP1
    impl 'paypay-lib1'
    - CVE1 CRITICAL
    impl 'paypay-lib2'
    impl 'com.foo:foo-bar'
    BOM: 2.3.6-PP2
    impl 'paypay-lib1'
    - CVE1
    impl 'paypay-lib2'
    impl 'com.foo:foo-bar'
    Upgrade dependencies included in BOM
    fix CVE1
    Minor Changes (1)
    

    View Slide

21. BOM: 2.3.6-PP1
    impl 'paypay-lib1'
    impl 'paypay-lib2'
    impl 'com.foo:foo-bar'
    BOM: 2.3.6-PP2
    impl 'paypay-lib1
    impl 'paypay-lib2'
    impl 'com.foo:foo-bar'
    impl 'paypay-lib3'
    Additional new dependencies to the BOM that do not require an
    upgrade to the Spring-Boot version.
    add
    paypay-lib3:1.0.0
    Minor Changes (2)
    

    View Slide

22. Final format:
    paypay-spring-bom:${SpringBootVersion}-PPX
    

    View Slide

23. How to create a BOM using gradle
    ● Use Java Platform Plugin
    ○ constraints, api to limit the version of a dependency
    ○ platform to source other BOMs.
    ○ Declare myPlatform in the publishing task to publish the artifact.
    dependencies {
    constraints {
    api 'commons-httpclient:commons-httpclient:3.1'
    api platform("org.springframework.boot:spring-boot-dependencies:2.3.11.RELEASE")
    }
    }
    publishing {
    publications {
    myPlatform(MavenPublication) {
    from components.javaPlatform
    }
    }
    }
    

    View Slide

24. BoM project structure
    ● Multi-module structure
    ○ A parent build.gradle file at root level controls all the version information for each BOMs.
    ○ Under each BOM folder there’s a build.gradle file for respective BOMs.
    project
    |--/boot-2.3
    | |--/build.gradle // for 2.3 BOM
    |--/boot-2.4
    | |--/build.gradle // for 2.4 BOM
    |
    |--...
    |--build.gradle
    

    View Slide

25. ● The root level build.gradle contains the following for each BOM:
    ○ Single dependency constraints.
    ○ Platform source versions and published version.
    // keyword: api
    dependentMap = [
    'boot-2.3': [
    'org.mybatis.spring.boot:mybatis-spring-boot-starter:2.1.4', // fix CVE
    'org.hibernate.validator:hibernate-validator:6.1.7.Final', // fix CVE
    'com.alibaba:fastjson:1.2.76'
    ],
    'boot-2.4': [ ... ]
    ]
    // keyword: platform
    versions = [
    'boot-2.3': [
    'paypaySpringBomVersion': '2.3.11-PP2', // published version
    'springbootVersion': '2.3.11.RELEASE', // platform source version
    ],
    'boot-2.4': [ ... ]
    ]
    project
    |--/boot-2.3
    | |--/build.gradle
    |--/boot-2.4
    | |--/build.gradle
    |
    |--...
    |--build.gradle
    

    View Slide

26. // do this to all subprojects
    subprojects {
    javaPlatform {
    allowDependencies()
    }
    dependencies {
    constraints {
    // check for subproject.name
    if (dependentMap.containsKey(project.name)) {
    dependentMap[project.name].each { dep -> api(dep) }
    }
    }
    }
    }
    project
    |--/boot-2.3
    | |--/build.gradle
    |--/boot-2.4
    | |--/build.gradle
    |
    |--...
    |--build.gradle
    

    View Slide

27. ● Apply the version constraints to each BoM
    version = "${paypaySpringBomVersion}" // published version
    dependencies {
    ...
    api platform("org.springframework.boot:spring-boot-dependencies:${springbootVersion}")
    api platform("org.springframework.cloud:spring-cloud-dependencies:${springbootVersion}")
    }
    project
    |--/boot-2.3
    | |--/build.gradle
    |--/boot-2.4
    | |--/build.gradle
    |
    |--...
    |--build.gradle
    

    View Slide

28. BOM verification
    ● Integration test
    ○ To catch regression.
    ○ To catch transitive dependency problem (this will happen).
    ○ Testing are shared among teams.
    ■ @EnableScheduling
    R
    |-- B:1.0.0 // newer Spring Boot doesn’t include this
    |-- C:1.0.0
    

    View Slide

29. View Slide

30. BoM migration
    ● Goal: Services migrate to BOM and remove vulnerabilities and make it easier
    to remove new ones.
    ● Two phases:
    ○ Front-facing Service:
    ■ Services that connect to outside world directly.
    ■ API gateways, third-party connections.
    ○ Backend services:
    ■ All Java services.
    

    View Slide

31. - Reduce 70% percentage of
    vulnerabilities.
    - Finish migration on ~180
    services in a quarter.
    - Finished all front-facing
    services.
    - Finished all backend services.
    BoM migration: Result
    Backend migration
    Front-end migration
    

    View Slide

32. Q & A
    Thank you !
    

    View Slide