Securing and managing dependencies at scale Ho YuehHsun

Ho YuehHsun - Joined PayPay at 2019/08. - DevSecOps team. - Focus on Dev side security. - Interests: Travel in Japan.

- Reduce 70% percentage of vulnerabilities. - Finish migration on ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. Reduce vulnerabilities at scale Backend migration Front-end migration

● A vulnerability is a bug which can be exploited by an attacker. ○ Known vulnerabilities is one of the most critical cyber risks. ○ CVEs (Common Vulnerabilities and Exposure) identify and catalog vulnerabilities. ■ Severity level: Critical, High, Medium, Low ○ An attack can facilitate data loss or server takeover. ■ Equifax breach in 2017 ○ Known vulnerabilities have exploit kits. Vulnerabilities and Exploits

● How to prevent: ○ Reducing threat surface. ■ Remove unneeded dependencies, components, files, and documentation. ○ Continuously monitor CVEs for vulnerabilities in components. ○ Fix or upgrade underlying platforms, frameworks, and dependencies regularly. Vulnerabilities and Exploits

- Micro-Service architecture - Services built from many 3rd-party and open-source dependencies. PayPay’s infrastructure Payment Finnet Wallet App POS Online ©PayPay Corporation.All rights reserved. CLM ...

● Upgrade the version in gradle build file ○ Teams doesn’t always know the best way to fix a vulnerability. ■ One by one, each by each ■ Resolution Strategy (Gradle) impl 'com.fasterxml.jackson.core:jackson-core:2.12.2' // previously 2.12.0 How to upgrade to a non-vulnerable version configurations.all { resolutionStrategy.eachDependency { DependencyResolveDetails details -> if (details.requested.group == 'org.apache.tomcat.embed') { details.useVersion '8.5.60' details.because 'Security Fixes (Multiple Issues)' } } }

Dependency Management in build system ● Maven ○ BOM (Bills of Materials): ■ BOM is a kind of POM file that is used to suggest the versions of a projects dependencies and provide a central place to define and update those versions. ■ The version can also be overridden.

spring-boot-dependencies BOM: ... org.springframework.boot spring-boot-test 2.3.11.RELEASE org.springframework.boot spring-boot-test-autoconfigure 2.3.11.RELEASE org.springframework.boot spring-boot-actuator 2.3.11.RELEASE org.springframework.boot spring-boot-actuator-autoconfigure 2.3.11.RELEASE ...

● Import a BOM in Maven ● Don’t need to declare the version if the BOM is imported org.springframework.boot spring-boot-starter org.springframework.boot spring-boot-starter-web org.springframework.boot spring-boot-dependencies 2.3.11.RELEASE pom import

Dependency Management in build system - Gradle - Dependency Management Plugin - A plugin created by the Gradle team to implement Maven’s dependency management. plugins { id 'org.springframework.boot' version '2.3.11.RELEASE' } ext { springBootVersion = ‘2.3.11.RELEASE’ } dependencyManagement { imports { mavenBom('org.springframework.boot:spring-boot-dependencies:${springBootVersion}') } }

● Gradle Platform ○ A platform is a software component in gradle system which can be used to control, suggest, enforce dependency versions. ○ Maven BOM is also a kind of platform that gradle supports. ext { springBootVersion = ‘2.3.11.RELEASE’ } implementation platform(“org.springframework.boot:spring-boot-dependencies:${springBootVersion}”)

Solution: BOM ● A dependency management artifact that allows us to control the versions for a set of core dependencies. ● Advantages: ○ Reduce the burden for team to maintain dependencies individually. ○ Define non-vulnerable versions of dependencies. ○ Introduce consistency of dependencies across services. ○ Version can be overridden.

Solution: BOM impl ('jp.ne.paypay:paypay-lib1:2.3.5') - CVE1, CVE2 impl ('jp.ne.paypay:paypay-lib2:4.1.7') - CVE3 impl ('com.foo:foo-bar:1.0.1') impl platform('paypay-bom:1.0.0') impl ('jp.ne.paypay:paypay-lib1') (0 CVE) impl ('jp.ne.paypay:paypay-lib2') (0 CVE) impl ('com.foo:foo-bar') PayPay BOM - 'jp.ne.paypay:paypay-lib1:2.4.0' CVE1, CVE2 - 'jp.ne.paypay:paypay-lib2:4.2.0' CVE3 - 'com.foo:foo-bar:1.0.1'

Solution: BOM PayPay BOM lib1:1.0.0 - Fix CVE1 CRITICAL lib2:1.1.0 - Fix CVE2 CRITICAL lib3:2.1.1 - Fix CVE3 CRITICAL … libX:2.0.0 - Fix CVEX LOW Payment Finnet Wallet POS Online ©PayPay Corporation.All rights reserved. CLM ...

Create the BOM ● Preliminary work: ○ Build System: gradle or maven? ○ Dependency selection. ○ Dependency constraints selection. ○ Naming and versioning.

○ Spring Boot and Spring Cloud dependencies based. ■ Spring Boot: 2.3.X, 2.4.X. ■ Spring Cloud: corresponding versions. ○ Internal PayPay Libraries. ○ Internal PayPay BOMs. ○ Fixed version of vulnerable dependencies that is critical or high severity level. Create the BOM: dependency selection

● Dependency constraints selection ○ Vulnerable dependencies: ■ The minimum version constraints that could fix the vulnerability. ■ Newest version is preferred. com.lib:lib.bar:1.2.0 - CVE 1 - CVE 2 com.lib:lib.bar:1.2.1 - CVE 1 - CVE 2 com.lib:lib.bar:1.3.0 - CVE 1 - CVE 2 ✅ Create the BOM: dependency constraint selection

○ Name: paypay-spring-bom ○ Version: ■ Format: ■ Use Spring-Boot version as part of the BOM version. ● Developers can easily determine the Spring-Boot version. ● example: ■ Add a patch number: PPX to the version indicating minor changes. ● X is a number and increased by 1 when there’s a minor change. ${SpringBootVersion}-PPX Create the BOM: naming and versioning paypay-spring-bom:2.3.6

BOM: 2.3.6-PP1 impl 'paypay-lib1' - CVE1 CRITICAL impl 'paypay-lib2' impl 'com.foo:foo-bar' BOM: 2.3.6-PP2 impl 'paypay-lib1' - CVE1 impl 'paypay-lib2' impl 'com.foo:foo-bar' Upgrade dependencies included in BOM fix CVE1 Minor Changes (1)

BOM: 2.3.6-PP1 impl 'paypay-lib1' impl 'paypay-lib2' impl 'com.foo:foo-bar' BOM: 2.3.6-PP2 impl 'paypay-lib1 impl 'paypay-lib2' impl 'com.foo:foo-bar' impl 'paypay-lib3' Additional new dependencies to the BOM that do not require an upgrade to the Spring-Boot version. add paypay-lib3:1.0.0 Minor Changes (2)

Final format: paypay-spring-bom:${SpringBootVersion}-PPX

How to create a BOM using gradle ● Use Java Platform Plugin ○ constraints, api to limit the version of a dependency ○ platform to source other BOMs. ○ Declare myPlatform in the publishing task to publish the artifact. dependencies { constraints { api 'commons-httpclient:commons-httpclient:3.1' api platform("org.springframework.boot:spring-boot-dependencies:2.3.11.RELEASE") } } publishing { publications { myPlatform(MavenPublication) { from components.javaPlatform } } }

BoM project structure ● Multi-module structure ○ A parent build.gradle file at root level controls all the version information for each BOMs. ○ Under each BOM folder there’s a build.gradle file for respective BOMs. project |--/boot-2.3 | |--/build.gradle // for 2.3 BOM |--/boot-2.4 | |--/build.gradle // for 2.4 BOM | |--... |--build.gradle

● The root level build.gradle contains the following for each BOM: ○ Single dependency constraints. ○ Platform source versions and published version. // keyword: api dependentMap = [ 'boot-2.3': [ 'org.mybatis.spring.boot:mybatis-spring-boot-starter:2.1.4', // fix CVE 'org.hibernate.validator:hibernate-validator:6.1.7.Final', // fix CVE 'com.alibaba:fastjson:1.2.76' ], 'boot-2.4': [ ... ] ] // keyword: platform versions = [ 'boot-2.3': [ 'paypaySpringBomVersion': '2.3.11-PP2', // published version 'springbootVersion': '2.3.11.RELEASE', // platform source version ], 'boot-2.4': [ ... ] ] project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle

// do this to all subprojects subprojects { javaPlatform { allowDependencies() } dependencies { constraints { // check for subproject.name if (dependentMap.containsKey(project.name)) { dependentMap[project.name].each { dep -> api(dep) } } } } } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle

● Apply the version constraints to each BoM version = "${paypaySpringBomVersion}" // published version dependencies { ... api platform("org.springframework.boot:spring-boot-dependencies:${springbootVersion}") api platform("org.springframework.cloud:spring-cloud-dependencies:${springbootVersion}") } project |--/boot-2.3 | |--/build.gradle |--/boot-2.4 | |--/build.gradle | |--... |--build.gradle

BOM verification ● Integration test ○ To catch regression. ○ To catch transitive dependency problem (this will happen). ○ Testing are shared among teams. ■ @EnableScheduling R |-- B:1.0.0 // newer Spring Boot doesn’t include this |-- C:1.0.0

No content

BoM migration ● Goal: Services migrate to BOM and remove vulnerabilities and make it easier to remove new ones. ● Two phases: ○ Front-facing Service: ■ Services that connect to outside world directly. ■ API gateways, third-party connections. ○ Backend services: ■ All Java services.

- Reduce 70% percentage of vulnerabilities. - Finish migration on ~180 services in a quarter. - Finished all front-facing services. - Finished all backend services. BoM migration: Result Backend migration Front-end migration

Q & A Thank you !