[OSS 2022] Improving CI/CD Process for Cloud Native Python Applications with PyPI Cloud

Python remains one of the most popular languages for building data science applications given its simplicity and widespread support for open-source data science projects. While it is easy to integrate such projects into your own Python applications, the CI/CD pipeline of the applications will become heavier and more tightly coupled, even in a modern containerized infrastructure. PyPI Cloud is an open-source private PyPI server which enables users to store and distribute their own Python packages in a cloud-native way. Not only does it provide easy-to-use package management similar to public PyPI, it also helps users build flexible and manageable Python applications by decoupling the CI/CD process of Python packages from application containerization. This talk explores how you can design cloud infrastructure that allows developers to focus on writing interesting Python code without worrying about how to build, deploy, or distribute it. By using a real world example, we will understand strategies to improve each step of the CI/CD pipeline for your Python applications using PyPI cloud.

Transcript

1. Improving CI/CD Process for Cloud Native Python Applications with PyPI Cloud Jaehyun Sim | Ikigai #ossummit @_simjay
2. #ossummit Introduction Jaehyun Sim Boston, MA Linkedin: linkedin.com/in/simjay paulsimsjh@gmail.com simjay@ikigailabs.io @ Ikigai since 2018 Ikigai Labs ikigailabs.io
3. #ossummit Motivation As a Tech startup, 1. We did not spend much time on our cloud infrastructure and CI/CD process 2. We spent lots of time trying too many options
4. #ossummit Overview • Where We Started • CI/CD Challenges • PyPI: The Python Package Index • Cloud Architecture with PyPI
5. Where We Started
6. #ossummit Where We Started To build a data science platform on cloud, we wanted to setup a microservice ecosystem with Python … and CI/CD pipeline for the services
7. #ossummit Continuous Integration (CI) Write Python code Compile Protobuf Run unittest Merge to main branch Build Docker Image
8. #ossummit Continuous Deployment (CD) Deploy to development Kubernetes cluster Deploy to production Kubernetes cluster Run platform tests against the development cluster Run platform tests against the production cluster (written in Python) (written in Python)
9. #ossummit CI/CD for Python Applications: “what we expected” API Server ETL Service ML Service
10. CI/CD Challenges
11. #ossummit Challenge: Shared Codebase
12. #ossummit Challenge: Shared Codebase -> Inefficiency API Server ETL Service ML Service Changes to the shared codebase trigger entire pipeline
13. #ossummit Decoupling Codebase
14. #ossummit Libraries in Python Module: A bunch of related code saved in a file with the extension `.py` Package: A directory of a collection of modules. To be considered a package, a directory must contain a file named `__init__.py` Library: A collection of packages. A library is an umbrella term referring to a reusable chunk of code.
15. #ossummit Is This Enough?
16. #ossummit Challenge: Tightly Coupled CI/CD Process For Services
17. #ossummit Challenge: Tightly Coupled CI/CD Process For Libraries
18. #ossummit Why Is This Bad? Problems 1. We still need to re-deliver services when library update is required 2. Hard to keep track of library versions within each service versions -> Decoupling is still required at CI/CD level
19. #ossummit Python Package Repository
20. #ossummit CI/CD for Libraries
21. #ossummit CI/CD for Services ml_model==0.0.2 ml_model==0.0.2 ml_model==0.0.2
22. #ossummit Service - Library Versioning ML Service ETL Service
23. #ossummit Are We Done? We can decouple CI/CD process of libraries and services “if” we have a Python package repository
24. The Python Package Index
25. #ossummit “Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.” - Cloud Native Computing Foundation (https://github.com/cncf/foundation/blob/main/charter.md) Cloud Native?
26. #ossummit Requirements for Python Package Repository 1. Portability 2. Security 3. Resiliency 4. Speed
27. #ossummit PyPI: The Python Package Index
28. #ossummit Public PyPI 1. Security: NONE 2. Resiliency: Yes 3. Portability: Yes 4. Speed: Yes
29. #ossummit PyPI Server
30. #ossummit PyPI Server OR
31. #ossummit PyPI Server: Scaling Issue? Reference: https://pypi.org/project/pypiserver/#serving-thousands-of-packages PyPI Server scans the entire package directory for every HTTP request -> This causes significant slow-downs when serving thousands of packages -> Can speed up with Caching with - pip install pypiserver[cache] - Enabling caching at reverse proxy
32. #ossummit PyPI Server - Demo https://github.com/simjay/pypi-demo/tree/main/pypi-server
33. #ossummit PyPI Cloud
34. #ossummit PyPI Cloud Portable!
35. #ossummit PyPI Cloud - Demo https://github.com/simjay/pypi-demo/tree/main/pypi-cloud
36. #ossummit PyPI Server vs PyPI Cloud [PyPI Server] 1. Portability: Yes 2. Security: Yes 3. Resiliency: Yes (with effort) 4. Speed: Yes (with effort) PyPI Server: Disk (NFS for Resiliency) Install manually containerized PyPI Cloud: Hosted Solution Hosted Solution containerized [PyPI Cloud] 1. Portability: Yes 2. Security: Yes 3. Resiliency: Yes 4. Speed: Yes
37. Cloud Architecture with PyPI
38. #ossummit Cloud Architecture with PyPI Server
39. #ossummit Cloud Architecture with PyPI Server
40. #ossummit Cloud Architecture with PyPI Cloud
41. In Conclusion…
42. #ossummit In Conclusion, By using PyPI cloud, we can 1. Organize codebase with libraries and services without sacrificing efficiency of CI/CD pipeline 2. Host scalable and secure Python Package Repository for your cloud native environment 3. Engineers can focus on building interesting stuff!
43. Thank you!