Improving CI/CD Process for Cloud Native Python Applications with PyPI Cloud Jaehyun Sim | Ikigai #ossummit @_simjay 

#ossummit Introduction Jaehyun Sim Boston, MA Linkedin: linkedin.com/in/simjay [email protected] [email protected] @ Ikigai since 2018 Ikigai Labs ikigailabs.io 

#ossummit Motivation As a Tech startup, 1. We did not spend much time on our cloud infrastructure and CI/CD process 2. We spent lots of time trying too many options 

#ossummit Overview • Where We Started • CI/CD Challenges • PyPI: The Python Package Index • Cloud Architecture with PyPI 

Where We Started 

#ossummit Where We Started To build a data science platform on cloud, we wanted to setup a microservice ecosystem with Python … and CI/CD pipeline for the services 

#ossummit Continuous Integration (CI) Write Python code Compile Protobuf Run unittest Merge to main branch Build Docker Image 

#ossummit Continuous Deployment (CD) Deploy to development Kubernetes cluster Deploy to production Kubernetes cluster Run platform tests against the development cluster Run platform tests against the production cluster (written in Python) (written in Python) 

#ossummit CI/CD for Python Applications: “what we expected” API Server ETL Service ML Service 

CI/CD Challenges 

#ossummit Challenge: Shared Codebase 

#ossummit Challenge: Shared Codebase -> Inefficiency API Server ETL Service ML Service Changes to the shared codebase trigger entire pipeline 

#ossummit Decoupling Codebase 

#ossummit Libraries in Python Module: A bunch of related code saved in a file with the extension `.py` Package: A directory of a collection of modules. To be considered a package, a directory must contain a file named `__init__.py` Library: A collection of packages. A library is an umbrella term referring to a reusable chunk of code. 

#ossummit Is This Enough? 

#ossummit Challenge: Tightly Coupled CI/CD Process For Services 

#ossummit Challenge: Tightly Coupled CI/CD Process For Libraries 

#ossummit Why Is This Bad? Problems 1. We still need to re-deliver services when library update is required 2. Hard to keep track of library versions within each service versions -> Decoupling is still required at CI/CD level 

#ossummit Python Package Repository 

#ossummit CI/CD for Libraries 

#ossummit CI/CD for Services ml_model==0.0.2 ml_model==0.0.2 ml_model==0.0.2 

#ossummit Service - Library Versioning ML Service ETL Service 

#ossummit Are We Done? We can decouple CI/CD process of libraries and services “if” we have a Python package repository 

The Python Package Index 

#ossummit “Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.” - Cloud Native Computing Foundation (https://github.com/cncf/foundation/blob/main/charter.md) Cloud Native? 

#ossummit Requirements for Python Package Repository 1. Portability 2. Security 3. Resiliency 4. Speed 

#ossummit PyPI: The Python Package Index 

#ossummit Public PyPI 1. Security: NONE 2. Resiliency: Yes 3. Portability: Yes 4. Speed: Yes 

#ossummit PyPI Server 

#ossummit PyPI Server OR 

#ossummit PyPI Server: Scaling Issue? Reference: https://pypi.org/project/pypiserver/#serving-thousands-of-packages PyPI Server scans the entire package directory for every HTTP request -> This causes significant slow-downs when serving thousands of packages -> Can speed up with Caching with - pip install pypiserver[cache] - Enabling caching at reverse proxy 

#ossummit PyPI Server - Demo https://github.com/simjay/pypi-demo/tree/main/pypi-server 

#ossummit PyPI Cloud 

#ossummit PyPI Cloud Portable! 

#ossummit PyPI Cloud - Demo https://github.com/simjay/pypi-demo/tree/main/pypi-cloud 

#ossummit PyPI Server vs PyPI Cloud [PyPI Server] 1. Portability: Yes 2. Security: Yes 3. Resiliency: Yes (with effort) 4. Speed: Yes (with effort) PyPI Server: Disk (NFS for Resiliency) Install manually containerized PyPI Cloud: Hosted Solution Hosted Solution containerized [PyPI Cloud] 1. Portability: Yes 2. Security: Yes 3. Resiliency: Yes 4. Speed: Yes 

Cloud Architecture with PyPI 

#ossummit Cloud Architecture with PyPI Server 

#ossummit Cloud Architecture with PyPI Server 

#ossummit Cloud Architecture with PyPI Cloud 

In Conclusion… 

#ossummit In Conclusion, By using PyPI cloud, we can 1. Organize codebase with libraries and services without sacrificing efficiency of CI/CD pipeline 2. Host scalable and secure Python Package Repository for your cloud native environment 3. Engineers can focus on building interesting stuff! 

Thank you! 

No content