Green IO Conference at apidays Paris 2024 - Track your ML and AI C02 Emissions with Code Carbon, Benoît Courty and Marion Coutarel, CodeCarbon

Track your ML and AI C02 Emissions with Code Carbon
Benoît Courty, President at CodeCarbon
Marion Coutarel, Contributor at CodeCarbon

Green IO Conference
apidays Paris 2024 - The Future API Stack for Mass Innovation
December 3 - 5, 2024

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. CodeCarbon Measuring the carbon footprint of Software Green IO Paris, December, 4th 2024
2. The Presenters Data Analyst and contributor to CodeCarbon MARION COUTAREL Data Scientist, French Parliament President and main contributor of Code Carbon BENOIT COURTY
3. 1. Introduction 2. CodeCarbon Today’s Program 3. Demo 4. 5. Future Work What can you do ?
4. 1. Introduction
5. What is CodeCarbon ? • A Python package • Why CodeCarbon ? Born in an AI lab… ◦ Old vision but critical today ◦ Get an idea of the orders of magnitude ◦ Comparison points • Real -time measurement while the code is running • Measures of the electricity consumption of the hardware
6. 2019: M L C 02, online tool to quantify M L emissions by researchers from M IL A Research Institute 2020 : C odeC arbon launch, open source python package to measure any code (not just M L ) emissions 2021 : Development of the C odeC arbon API and Dashboard with Data For Good 2023 : C reation of a non-profit in France to support the project 2024 : Add a security layer and rebuild the dashboard A bit of History
7. How does it work ?
8. Carbon Intensity of Electricity Source Electricity Map
9. Inference is the new problem Carbon emissions of one query (about 400 output tokens) Tool Emissions in grams of CO2 Google Search 0,2 Llama 3 8B 0,9 GPT-4o 22 Drive 1 km in fuel car 140 GPT-4 270 Claude 3 Opus 330 Source : https://huggingface.co/spaces/genai-impact/ecologits-calculator EcoLogits
10. 2. Using CodeCarbon
11. We want it simple > pip install codecarbon > codecarbon config > codecarbon monitor from codecarbon import track_emissions @track_emissions() def my_new_ai_model(request): return “I'm sorry, but as an AI, I can’t answer your question” OR Yes, it’s simple for developers ! (venv - codecarbon) codecarbon@ben - tower:~$ codecarbon monitor -- no- api [codecarbon INFO @ 22:19:42] >>> Tracker's metadata: [codecarbon INFO @ 22:19:42] Available RAM : 31.176 GB [codecarbon INFO @ 22:19:42] CPU model: AMD Ryzen Threadripper 1950X 16 - Core Processor [codecarbon INFO @ 22:19:42] GPU model: 1 x NVIDIA GeForce GTX 1080 Ti [codecarbon INFO @ 22:19:45] Saving emissions data to file /home/codecarbon/emissions.csv [codecarbon INFO @ 22:19:55] Energy consumed for RAM : 0.000032 kWh. RAM Power : 11 W [codecarbon INFO @ 22:19:55] Energy consumed for all GPUs : 0.000652 kWh. Total GPU Power : 234 W [codecarbon INFO @ 22:19:55] Energy consumed for all CPUs : 0.000920 kWh. Total CPU Power : 330 W [codecarbon INFO @ 22:19:55] 0.001604 kWh of electricity used since the beginning.
12. 3. Quick Demo
13. Follow Along Google Colab
15. Hardware Install PyPI, Conda Hatch for env and packaging Running CLI (no Python code) Context manager Decorator Outputs CSV CodeCarbon API Webhook LogFire Prometheus Support GPU (Nvidia and soon AMD ROC) CPU Intel and AMD via RAPL, Powermetrics, TDP database RAM based on a constant No disk
16. 5. Conclusion
17. As a front developer As a data scientist Use finetuned models Location and time of trainings Use optimization tools like Zeus Keep hardware longer As a user Use AI systems responsibly - Smallest model for your need Keep you hardware longer In IT Operations Pick hosting location according to carbon intensity Keep hardware longer Reducing Code’s emissions Build lighter websites Pause C opilot when not needed Keep hardware longer
18. Secure API and Dashboard , making CodeCarbon easier to adopt in enterprise setting : - Migrating the dashboard from plotly dash, to a React Web App with authentication system using Fief Build partnerships with observability platforms (prometheus, logfire) to integrate into existing monitoring Keep presenting CodeCarbon to recruit users and developers Future Work
19. Thanks ! See you on https://github.com/mlco2/codecarbon ! Data For Good Mozilla Tech Fund 2024
20. BONUS
21. CodeCarbon was selected into the 2024 Mozilla Tech Fund ! This allows us to invest in developments to go faster, by hiring freelancers This allows us to finance conferences like this one where we can go and advocate for a responsible use of AI Mozilla Tech Fund 2024
22. We have very few disclosed metrics, mostly only for Llama: - L lama 2 70B : 300 tC O2eq (for 2 Trillions training tokens) - L lama 3 70B : 1 900 tC O2eq (for 15 Trillions training tokens) - L lama 3 405B : 8 930 tC O2eq (for 15 Trillions training tokens) 1 tC O2eq ⇔ 5 000 km with a fuel car or 15 tonnes of ice melted forever on earth - Biggest public datacenter in France, after 2024 upgrade : 3 704 GPUs - 4 M W - Elon M usk Colossus , GPU cluster : 100,000 GPUs - 150 MW AI to Mars : Bigger and bigger
24. What it is not ? • Only the real -time usage, not the build of the hardware or the end of life. • We measure electricity and estimate emission, not water usage • Do not tell you how to modify your code