Measuring the Power Consumption and Carbon Emissions of Your Local LLM on Kubernetes

CNCF Cloud Native Sustainability Week 2024 – Local Meetup Tokyo!でLT登壇した発表資料
https://community.cncf.io/events/details/cncf-cloud-native-community-japan-presents-cncf-cloud-native-sustainability-week-2024-local-meetup-tokyo/

Transcript

1. Measuring the Power Consumption and Carbon Emissions of Your Local LLM on Kubernetes Akiho Miyamura CNCF Cloud Native Sustainability Week 2024 – Local Meetup Tokyo! 4/10/2024
2. Who am I? ・Akiho Miyamura ・Software Engineer ・ interested in GreenOps ・I gave a presentation introducing Kepler at Kubernetes Meetup Tokyo #58 KubeCon EU 2023 Recap - Title: Sustainability Through Accountability in a CNCF Ecosystemのご紹介
3. Power consumption of LLM ・the demand for electricity is increasing due to using of a generative AI(*1) - Global consuming estimated 460TWh in 2022 → exceed 1000TWh in 2026 This figure is comparable to the total electricity consumption of Japan ・When using services like ChatGPT, we rarely consider the electricity consumption or carbon emissions ・The use of local LLMs is increasing, so it's important to understand the electricity consumption and carbon emissions associated with local LLMs. ・How can we calculate it in the case of running a local LLM on k8s? (*1) https://www.iea.org/reports/electricity-2024/executive-summary
4. Kepler ・is CNCF Sandbox project ・is Kubernetes Efficient Power Level Exporter - to measure power consumption of workload by tracing cpu performance counter and linux kernel tracepoint ・uses eBPF to probe energy-related system stats and exports them as Prometheus metrics. ・can visualise Kepler metrics with Grafana
5. You only need to deploy kepler and copy kepler_dashboard.json to grafana Kepler
6. Kepler Dashboard of Grafana Carbon Emissions Total power consumption (Watt) of Pods in NS Total power consumption (PKG+DRAM+O THER+GPU) by NS (kWh/day)
7. PC Spec Mini PC: GMKtec nucbox3 (I bought it on Amazon for about 10,000 yen ) ・CPU: Celeron J4125 ・RAM: DDR4-2400 16GB SODIMM ・HDD: 250GB
8. Models used for Comparison ・Model 1 : llama 3.2 3.2b ・Model 2 : gemma2 2b ・Model 3 : gemma2 9b Benchmark question: I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. I also gave 3 bananas to my brother. How many apples did I remain with? Let's think step by step.(私は市場へ行 きリンゴを10個買いました。私はリンゴを2個隣の人に、もう2個を修理屋さんにあげました。そ れからリンゴをさらに5個買い、1個食べました。また、バナナを3本兄にあげました。リンゴは何 個残ったでしょう？段階的に考えてみましょう。) (*2) Measuring the Power Consumption and Carbon Emissions of each LLM models (*2)Asking 60+ LLMs a set of 20 questions(https://benchmarks.llmonitor.com/) The subject of measurement is ollama pod
9. Result of llama 3b
10. Result of gemma2 2b
11. Result of gemma2 9b
12. Results model CO2 coal (pounds/kWh/day) CO2 petroleum (pounds/kWh/day) CO2 natural gas (pounds/kWh/day) Power Consumption (kWh/day) llama3.2 3b 0.0722 0.0690 0.0295 0.0330 gemma2 2b 0.0401 0.0383 0.0164 0.0180 gemma2 9b 0.0573 0.0548 0.0234 0.0257 ① ② ① llama 3.2 > gemma2 9b ② gemma2 2b < gemma2 9b
13. Summary How to measure the Power Consumption and Carbon Emissions ・By using kepler, you can measure the consumption of LLM pod on kubernetes. You only need to deploy kepler and copy kepler_dashboard.json to grafana. the Power Consumption and Carbon Emissions by LLM models ・For the same model, a larger model size consumes more power. ・For different models, even a smaller model size can sometimes consume more power.
14. Only ollama ollama + open-webui Appendix: Result of ollama + open-webui
15. Appendix: Answer of llama3.2 3b
16. Appendix: Answer of gemma2 2b
17. Appendix: Answer of gemma2 9b
18. Appendix: Kubecost Kubecost has incorporated Carbon Costs(CO2 emissions), but it is not yet supported in On-prem Carbon Costs