[PyCon KR 2026] More Variants, More Diversity for AI Accelerators

Transcript

1. More Variants, More Diversity for AI Accelerators 김준기 (Joongi Kim) 래블업 주식회사 (Lablup Inc.) PyCon KR 2026
2. About Myself • CTO at Lablup Inc. (2016~current) ◦ • Serial seasoned PyCon speaker since 2015... ◦ • Leading development of the Backend.AI core platform & fractional GPU virtualization technology Talks in KR, APAC(KR, JP, SG), US, JP, TH Contributor in aio-libs, maintaining aiodocker & aiomonitor ◦ Also wrote aiotools & Callosum • Ph.D. in Computer Science, KAIST (High-perf packet processing framework for heterogeneous accelerators) • https://speakerdeck.com/achimnol • https://github.com/achimnol
3. Table of Contents • Recap: PEP 817 & 825 Wheel Variants (follow-up to my talk in PyCon KR 2025) ◦ Why can’t we just do “pip install torch”? ◦ Extending the wheel platform tags with vendor-provided variants ◦ The updates since last year • This talk: Could we bring the same idea to the container world? • Experiments & Applications ◦ Docker variant plugin ◦ Backend.AI platform integration
4. The Problem and Motivation Why can't we just do "pip install torch"? ✓ Platform tags can only handle OS type & CPU architectures and glibc versions (manylinux). ✓ There are multiple (increasing #) accelerator vendors, and each one may have its own exotic compatibility tables and rules.
5. Concept of Wheel Variants 3-tuple property sets defined as namespace :: feature :: value Source: https://labs.quansight.org/blog/python-wheels-from-tags-to-variants
6. Design of Wheel Variants Example: – Non-variant (bare) wheel: numpy-2.3.2-cp313-cp313t-musllinux_1_2_x86_64.whl – Wheel with variant label x86_64_v3: numpy-2.3.2-cp313-cp313t-musllinux_1_2_x86_64-x86_64_v3 .whl – Null variant: numpy-2.3.2-cp313-cp313t-musllinux_1_2_x86_64-null .whl Source: https://labs.quansight.org/blog/python-wheels-from-tags-to-variants
7. Updates in PEP 817 & PEP 825 • Major specification changes during 2025 → 2026 Topic 2025 (WheelNext proposal) 2026 (PEP 817 & PEP 825) Variant ID format Hash-based variant index Human-readable slug Security model Automatic loading & execution of variant providers Explicit opt-in with vendoring well-known providers Property model 3-tuple namespace :: feature :: value all-or-nothing (conjunctive) matching Same 3-tuple, extended with the multi-value property semantic that allows one-of-them matching, Range matching with commonly defined _lower_bound or _upper_bound suffixes “null” variant Zero number of properties are represented as the “00000000” variant. Same, represented as “null”, with an explicitly higher precedence than bare (non-variant) wheels Variant-specific dependencies (None) e.g., dep1; "nvidia" in variant_namespace in requirements with pylock.toml integration ABI support (None) Introduce a reserved namespace: abi_dependency (e.g., “abi_dependency :: torch :: 2.9”)
8. Packaging Summit 2026 • https://us.pycon.org/2026/events/packaging-summit/ • Meetup for Python packaging “stakeholders” ◦ Maintainers and developers of: ▪ ▪ • Package tooling such as setuptools, pip, conda, poetry, uv Difficult-to-distribute libraries such as numpy, torch, xgboost ◦ Downstream package maintainers from Linux distros ◦ All users who are interested in the topic Format: A half-day workshop ◦ 3~4 Selected topic talks followed by Lightning Talks and round-table discussions @pradyunsg (Pradyun Gedam) @jezdez (Jannis Leidel) @cam-gerlach (C.A.M. Gerlach)
9. Packaging Summit 2026: An excerpt • Revisiting Wheel 2.0 and Better Compression (Emma Smith) ◦ https://docs.google.com/presentation/d/1zh-3FkCg2cSMp3QD5oFji5sebeJlu3Hg_d-maYe7Pno/edi t?usp=sharing ◦ How to not break existing CI/CD pipelines after changing the wheel format? ◦ ▪ Minimal change via PEP 777 → sub-PEP per individual change ▪ Backward compatibility in wheel formats while index providers decide the transition timing. Adoption of Zstandard ▪ Expected savings: 25% size reduction of top 1000 projects, 100PB traffic, 36-years of decompression latencies ▪ Challenge: toolings like pip don’t like adding C extensions. • ▪ ◦ But, Python 3.14 included zstd as a stdlib. Proposal: Store tar.zst uncompressed inside ZIP for backward compatibility QnA ▪ Whether to allow other compression formats like xz? ▪ Uncompressed metadata for tools like uv? ▪ Conda’s “zip streaming writes” idea?
10. How is it going in the container world? • Platfrom variants in OCI (Open Container Image) ◦ Borrows Golang's build target variant expressions ◦ Several discussions related to expand variant specs (opencontainers/image-spec#1216) ☹ No consideration for accelerators Source: https://specs.opencontainers.org/image-spec/image-index/#platform-variants
11. How is it going in the container world? • Like many other systems, Backend.AI has adopted a custom tagging ruleset. ◦ pytorch:2.12.0-ubuntu24.04-py312 ◦ pytorch:2.12.0-ubuntu24.04-cuda13-py312 ◦ pytorch:2.12.0-ubuntu22.04-atom-py313 ◦ pytorch:2.12.0-ubuntu22.04-... ☹ Manual client-side selection (exact, partial, ...?) ☹ No standardized tag namings across different vendors ☹ Too long tags for multi-feature-compatible images
12. State-of-the-art? • K8s DRA (Dynamic Resource Allocation) [stable in v1.35] • Prioritized List [stable in v1.35] & DRAListTypeAttributes [alpha in v1.36] ◦ Consumed by CEL expressions to generalize matching conditions ◦ Specifies alternative combinations of device properties. ◦ Updated DRAListTypeAttributes introduces typed values (bools, ints, strings, versions). ☹ Looks promising, but complexity still left to users (Manual CEL expression writing...) Source: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/
13. Same Analogy, Same Problem Python Package Index Package index with variant props Query package list pip install Download the matched wheel variant Variant props from local variant providers Client Host CPU Features GPU Features NPU Features x86_64 :: level :: v3 nvidia :: cuda_version :: 12.8 nvidia :: sm_arch :: 120_real rebellions :: arch :: atom_max furiosa :: arch :: renegade hyperaccel :: arch :: bertha
14. Same Analogy, Same Problem Python Package Index Container Registry Package index with variant props Image index with variant props Query package list pip install Download the matched wheel variant Variant props from local variant providers Client Host CPU Features GPU Features NPU Features Query image list docker pull Pull the matched container variant Variant props from local variant providers Client Host x86_64 :: level :: v3 nvidia :: cuda_version :: 12.8 nvidia :: sm_arch :: 120_real rebellions :: arch :: atom_max furiosa :: arch :: renegade hyperaccel :: arch :: bertha CPU Features GPU Features NPU Features x86_64 :: level :: v3 nvidia :: cuda_version :: 12.8 nvidia :: sm_arch :: 120_real rebellions :: arch :: atom_max furiosa :: arch :: renegade hyperaccel :: arch :: bertha
15. Experiment: Docker Variant Plugin • https://github.com/lablup/docker-variant .variant.index docker CLI .variant.labels OCI-compatible Registry .variant.ordering .docker.client .providers docker variant plugin $ docker variant push ... $ docker variant pull ... • • .oci.client .variant.property variant-proxy .oci.index $ variant-proxy --upstream registry.internal:5000 Two components ◦ Client-side docker variant CLI plugin ◦ Server-side container registry proxy that adds /v2/<name>/_variants/<version> API Currently implemented provider plugins ◦ nvidia (based on nvidia-smi CLI command) ◦ x86_64, aarch64 (based on feature detectors from golang.org/x/sys/cpu)
16. Experiment: Docker Variant Plugin • Building a variant-specific container image Dockerfile You may parametrize labels and build steps using ARG directives. ... LABEL dev.pep817.variant-label="cu128" \ dev.pep817.variant.nvidia.cuda_version_lower_bound="12.8" ... Namespace Feature Value $ docker build -t registry.internal/myapp:1.0.0-cu128 . $ docker variant push registry.internal/myapp:1.0.0-cu128 • Using the variant image For local testing, use the proxy’s host:port when referring to the images. $ docker variant pull registry.internal/myapp:1.0.0 Selected variant "cu128" of registry.internal/myapp version 1.0.0 ... Tagged registry.internal/myapp:1.0.0 and registry.internal/myapp:1.0.0-cu128 ...
17. Application: Motivation • Choose appropriate nodes having compatible accelerators for a given workload ◦ ◦ Different GPU generations come with different compute capability and SM architectures. Workload compatibility comes from the CUDA runtime version in container images. • Need to support multi-node jobs (aka gang scheduling) ◦ A single resource slot “cuda.device” or “cuda.shares” cannot distinguish various device-specific & workload-specific variants. ◦ All sub-containers within a workload should have identical/compatible hardware combinations across multiple nodes.
18. Application: Workload's variant spec is populated from container image labels (scanned & cached in prior) • Solution ◦ Per-node variant providers from device plugins → Agent-Device variant registry ◦ Per-workload variant properties based on image labels ◦ Variant matcher within the scheduler Source: https://github.com/lablup/backend.ai/pull/11404
19. Summary • • Wheel Variants: expands the platform tags to multi-vendor accelerator variants ◦ The specification is being actively updated and discussed. ◦ 2026 version includes more practical extensions like variant-specific dependencies. Docker Variant Plugin ◦ • • A simple extension to the existing container ecosystem (CLI plugin, registry proxy) Backend.AI ◦ Agent-Device registry table ◦ Reusing the variant data models Future Work ◦ Build and distribute variant-aware container images (Q: could we automate labeling?) ◦ We are looking for provider participation from AI chip vendors!