More Charts More Problems

Title: More charts, more problems

Package managers are hard. Helm learned a lot of lessons from others’ mistakes, but also repeated some. For example, having a single index file per repository is not scalable. It’s the same mistake that NPM made and it causes slower CI, high memory consumption, slower searches, and more. In this lightning talk, we’ll explore several solutions to the problem, such as compression and partitioning, their strengths, and their weaknesses

Transcript

1. www.jfrog.com Copyright © 2017 JFrog. All Rights Reserved More Charts More Problems Ankush Chadha – Senior Solution Developer Feb 22th 2018
2. 2 Copyright © 2017 JFrog. All Rights Reserved | # of charts vs memory docker run -m=512m -it alpine-helm helm repo add helm-prod $helm-repo
3. 3 Copyright © 2017 JFrog. All Rights Reserved | Questions to ask 1. Define scale 2. What is the problem? 3. How to reproduce the problem? 4. How to solve this problem?
4. 4 Copyright © 2017 JFrog. All Rights Reserved | More Charts # of charts per app ~ F(app, deployment topology, dependencies)
5. 5 Copyright © 2017 JFrog. All Rights Reserved | More Charts base-image:1.0 Manifest sha256:252564 .. OS: SHA2 Layer n: SHA2 Manifest sha256:462564. . ac-app:1.0 app: SHA2: 4 app containerized app topology 1 2 3 4
6. 6 Copyright © 2017 JFrog. All Rights Reserved | More Charts: Comparable - NPM 2015 2017 # of packages 12500 400,000 https://www.linux.com/news/event/Nodejs/2016/state-union-npm
7. 7 Copyright © 2017 JFrog. All Rights Reserved | Problems •Single index file per repository – Good to begin – But with scale (# of charts and # of devices) there are challenges - – Slow • Bootstrap - helm repo add … • Search • Merge – Question to ask • Metadata part of single index file
8. 8 Copyright © 2017 JFrog. All Rights Reserved | Helm data generator 115 charts 1000 versions per chart 115k charts https://github.com/ankushchadha/helm-chart-generator docker run -m=512m -it alpine-helm helm repo add helm-prod $helm-repo-url X
9. 9 Copyright © 2017 JFrog. All Rights Reserved | Solution (Proposal) •Popular Partitioning technique index.yaml (no metadata such as description, etc) – /chartName1/index.yaml (includes latest and versions) – /chartName2/index.yaml – /chartName3/index.yaml » Relative path vs absolute path