Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

AI'S Hidden cost: The environmental impact of M...

Luca Corbucci
November 10, 2024

AI'S Hidden cost: The environmental impact of Machine Learning

Luca Corbucci

November 10, 2024
Tweet

More Decks by Luca Corbucci

Other Decks in Science

Transcript

  1. Climate change is one of our genera ti on’s biggest

    challenges Why this talk? We are trying to fi ght it but at the same ti me we fi nd new ways to hurt our planet
  2. Climate change is one of our genera ti on’s biggest

    challenges Why this talk? We are trying to fi ght it but at the same ti me we fi nd new ways to hurt our planet We all love using AI Tools but what is the environmental impact?
  3. Climate change is one of our genera ti on’s biggest

    challenges Why this talk? We are trying to fi ght it but at the same ti me we fi nd new ways to hurt our planet We all love using AI Tools but what is the environmental impact? The news about this topic are not encouraging
  4. Why this talk? As a computer scien ti st, I

    asked myself what is the cost to the environment of using these tools
  5. Why this talk? As a computer scien ti st, I

    asked myself what is the cost to the environment of using these tools I started to read more about this topic
  6. Why this talk? As a computer scien ti st, I

    asked myself what is the cost to the environment of using these tools I started to read more about this topic Today I’ll tell you what I have learnt
  7. Luca Corbucci Podcaster @ PointerPodcast Community Manager @ SuperHero Valley

    Community Manager @ Pisa.dev https://lucacorbucci.me/ PhD Student in Computer Science @ University of Pisa - Responsible AI
  8. GPT 3 175 Billion parameters GPT-4 1.76 Trillion parameters (uno

    ffi cial) 2020 2023 Models are get ti ng bigger…
  9. GPT o1-preview 2.8 trillion parameters (uno ffi cial) GPT 3

    175 Billion parameters GPT-4 1.76 Trillion parameters (uno ffi cial) 2020 2023 2024 Models are get ti ng bigger…
  10. Most closed LLMs do not give any informa ti on

    about their size. It is hard to know what was the cost of the training
  11. There are some es ti ma ti ons for the

    training of GPT 3: 1,300 MWh of electricity and 552 tonnes of CO2 Most closed LLMs do not give any informa ti on about their size. It is hard to know what was the cost of the training
  12. GPT 3 175 Billion parameters 130 US homes in a

    year There are some es ti ma ti ons for the training of GPT 3: 1,300 MWh of electricity and 552 tonnes of CO2 Training ( ) = 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 🏠 Most closed LLMs do not give any informa ti on about their size. It is hard to know what was the cost of the training
  13. Trained for 1.08 million GPU hours BigScience Large Open- science

    Open-access Mul ti lingual Language Model (BLOOM) Trained on 348 A100 GPUs
  14. Trained for 1.08 million GPU hours BigScience Large Open- science

    Open-access Mul ti lingual Language Model (BLOOM) Trained on 348 A100 GPUs We can have an es ti mate of the energy used and of the carbon footprint of training an LLM
  15. Trained for 1.08 million GPU hours BigScience Large Open- science

    Open-access Mul ti lingual Language Model (BLOOM) Trained on 348 A100 GPUs We can have an es ti mate of the energy used and of the carbon footprint of training an LLM
  16. How can we measure the energy consump ti on? TDP

    = 400W (They considered 100% usage)
  17. How can we measure the energy consump ti on? TDP

    = 400W (They considered 100% usage) 348 GPUS X
  18. TDP = 400W (They considered 100% usage) 348 GPUS 1,082,990

    Hours X X How can we measure the energy consump ti on?
  19. TDP = 400W (They considered 100% usage) 348 GPUS 1,082,990

    Hours X X = 433.196 kWh How can we measure the energy consump ti on? Images from https://thenounproject.com/
  20. X Energy Consump ti on 433.196 kWh Carbon intensity of

    the energy grid 57 gCo_2eq/KWh How can we measure the carbon footprint?
  21. X Energy Consump ti on 433.196 kWh Carbon intensity of

    the energy grid 57 gCo_2eq/KWh 24.69 tonnes of CO2 How can we measure the carbon footprint? =
  22. 24.69 tonnes of CO2 Training ( ) = 101623 km

    driven by an average gasoline- powered passenger vehicle
  23. Inference (for LLMs) The LLM is loaded on the GPUs

    We give a prompt to the LLM and we wait for an answer
  24. Inference (for LLMs) The LLM is loaded on the GPUs

    The LLM predict a series of tokens based on the prompt We give a prompt to the LLM and we wait for an answer
  25. Inference (for LLMs) The LLM is loaded on the GPUs

    The LLM predict a series of tokens based on the prompt This phase is called inference and we need one or more GPUs We give a prompt to the LLM and we wait for an answer
  26. “A single Google search takes 0.3 watt-hours of electricity, while

    a ChatGPT request takes 2.9 watt- hours” AI already uses as much energy as a small country. It’s only the beginning. https://www.vox.com/climate/2024/3/28/24111721/climate-ai-tech-energy- demand-rising
  27. “If ChatGPT were integrated into the 9 billion searches done

    daily, the electricity demand would increase by 10 terawatt-hours a year — the amount consumed by about 1.5 million European Union residents. “ Electricity 2024: Analysis and forecast to 2026 - https://bit.ly/3C13ZZ3
  28. Deployed on a GCP instance with 16 GPUs Total consump

    ti on 914KWh of electricity 230k requests in 18 days Energy Consump ti on of training 433.196 kWh
  29. Deployed on a GCP instance with 16 GPUs CPU 2%

    GPU 75% RAM 23% Total consump ti on 914KWh of electricity The consump ti on is ∼0.28kWh even if the model is not answering to ques ti ons 230k requests in 18 days
  30. The training represents the biggest cost in terms of energy

    but inference could exceed training emissions in just a few weeks
  31. Inference of 88 Gen AI models compared Power Hungry Processing:

    Watts Driving the Cost of AI Deployment? https://arxiv.org/abs/2311.16863
  32. Inference of 88 Gen AI models compared 💡Image genera ti

    on is more costly than text genera ti on Power Hungry Processing: Watts Driving the Cost of AI Deployment? https://arxiv.org/abs/2311.16863
  33. Inference of 88 Gen AI models compared 💡Image genera ti

    on is more costly than text genera ti on 🚗 6.5 Km vs 🚗. 0.0009 Km Power Hungry Processing: Watts Driving the Cost of AI Deployment? https://arxiv.org/abs/2311.16863
  34. What is the main problem with inference and “famous” models

    Power Hungry Processing: Watts Driving the Cost of AI Deployment? https://arxiv.org/abs/2311.16863
  35. What is the main problem with inference and “famous” models

    For models like ChatGPT, 2 weeks could be enough to have higher inference cost than training Power Hungry Processing: Watts Driving the Cost of AI Deployment? https://arxiv.org/abs/2311.16863
  36. How can I choose an LLM while considering the energy

    e ff i ciency? Ml.Energy Leaderboard https://ml.energy/leaderboard/
  37. Data Centers It is not enough to op ti mize

    training and inference, we need to op ti mize the en ti re training stack
  38. Data centers power demand over ti me AI is poised

    to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  39. Data centers power demand over ti me Increased workload +

    Increased data centers e ff i ciency AI is poised to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  40. Data centers power demand over ti me AI is poised

    to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  41. Data centers power demand over ti me Increase in power

    demand and decrease in the e ffi ciency gain AI is poised to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  42. Data centers power demand over ti me AI is poised

    to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  43. Data centers power demand over ti me AI could represent

    the 30% of the data centers power demand in 2030 AI is poised to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  44. Data centers power demand over ti me The es ti

    mated increase of power consump ti on in Europe could be equal to the power consump ti on of Nederland, Greece and Portugal AI is poised to drive 160% increase in data center power demand https://www.goldmansachs.com/insights/ar ti cles/AI-poised-to-drive-160-increase- in-power-demand
  45. Is it only a matter of energy? Microsoft recently opened

    new datacenters in Goodyear, Arizona. Image from https://deepgram.com/learn/how-ai-consumes-water Making AI Less “Thirsty”: Uncovering and Addressing thr Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271
  46. Is it only a matter of energy? Microsoft recently opened

    new datacenters in Goodyear, Arizona. Not only do they consume energy but they also need a lot of water to keep a low temperature. Image from https://deepgram.com/learn/how-ai-consumes-water Making AI Less “Thirsty”: Uncovering and Addressing thr Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271
  47. Is it only a matter of energy? Image from https://deepgram.com/learn/how-ai-consumes-water

    Microsoft recently opened new datacenters in Goodyear, Arizona. Not only do they consume energy but they also need a lot of water to keep a low temperature. Training GPT-3 in Microsoft’s data centers can evaporate 700,000 liters of clean freshwater Making AI Less “Thirsty”: Uncovering and Addressing thr Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271
  48. Is it only a matter of energy? Asking between twenty

    and fi fty ques ti ons on ChatGPT is equivalent to consuming half a litre of water. Image from https://deepgram.com/learn/how-ai-consumes-water Making AI Less “Thirsty”: Uncovering and Addressing thr Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271
  49. Is it only a matter of energy? Asking between twenty

    and fi fty ques ti ons on ChatGPT is equivalent to consuming half a litre of water. Researchers at UC Riverside es ti mated that global AI demand could cause data centers to use more than 4 trillion liter of fresh water by 2027. Image from https://deepgram.com/learn/how-ai-consumes-water Making AI Less “Thirsty”: Uncovering and Addressing thr Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271
  50. The solu ti on they found They invested in two

    startups that produces small reactors. They are paying to revive the shuttered Three Mile Island nuclear power plant in Pennsylvania They have been stopped by 🐝🐝🐝🐝🐝
  51. Mi ti ga ti on Strategies Specialized Hardware Improve the

    E ff i ciency of the accelerators Hardware solu ti ons
  52. Mi ti ga ti on Strategies Specialized Hardware Improve the

    E ff i ciency of the accelerators Hardware solu ti ons Etched: An ASIC Specialized for Transformer based Model inference
  53. Fine-Tuning Quan ti za ti on Small language models New

    training Algorithms Mi ti ga ti on Strategies Specialized Hardware Improve the E ff i ciency of the accelerators Hardware solu ti ons Algorithmic Techniques
  54. Fine-tuning Llama 3.2 70B The big Genera ti ve Models

    try to do a lot of things at once This reduces the training cost but increases the inference Fine-tuning can create smaller models that are more specialized and consume less energy.
  55. Quan ti za ti on Llama 3.2 70B Modern transformer-based

    LLMs rely heavily on matrix mul ti plica ti on The quan ti za ti on converts the weights from high-precision values to lower-precision ones. Example: the weighs are converted from 32-bit fl oa ti ng-point number to an 8-bit integer Llama 3.2 70B
  56. Matmul free Language models Modern transformer- based LLMs rely heavily

    on matrix mul ti plica ti on opera ti ons Can we reduce the matrix mul ti plica ti on opera ti ons?
  57. Matmul free Language models Instead of using MatMul, they proposed

    to use addi ti ve opera ti ons and ternary weights. The goal is to have a compe ti ti ve performance while reducing resource demands.
  58. As a researcher I can report as much details as

    I can on the training details and energy cost Conclusion
  59. As a researcher I can report as much details as

    I can on the training details and energy cost As user of Gen AI Tools I can try to exploit as much as possible the on-device models Conclusion
  60. As a researcher I can report as much details as

    I can on the training details and energy cost As user of Gen AI Tools I can try to exploit as much as possible the on-device models As a developer that wants to implement Gen AI tools inside their app I could try to fi nd a balance between energy e ffi ciency and “u ti lity” of the tool Conclusion
  61. As a researcher I can report as much details as

    I can on the training details and energy cost As user of Gen AI Tools I can try to exploit as much as possible the on-device models As a developer that wants to implement Gen AI tools inside their app I would try to fi nd a balance between energy e ffi ciency and “u ti lity” of the tool I’m op ti mis ti c on this topic . Different companies are compe ti ng to create the best possible Gen AI. It is useful for them to have energy e ffi cient models. Conclusion
  62. References -AI/data centers' global power surge and the Sustainability impact

    https:// www.goldmansachs.com/images/migrated/insights/pages/gs-research/gs- sustain-genera ti onal-growth-ai-data-centers-global-power-surge-and-the- sustainability-impact/sustain-data-center-redac ti on.pdf -Making AI Less “Thirsty”: Uncovering and Addressing the Secret Water Footprint of AI Models https://arxiv.org/pdf/2304.03271 -AI Is Taking Water From the Desert https://www.theatlan ti c.com/technology/ archive/2024/03/ai-water-climate-microsoft/677602/ -Power Hungry Processing: Watts Driving the Cost of AI Deployment? https:// arxiv.org/abs/2311.16863 -MatMulfree LM https://huggingface.co/collec ti ons/ridger/matmulfree- lm-665f4d2b4e4648756e0dd13c -A Guide to Quan ti za ti on in LLMs https://symbl.ai/developers/blog/a-guide-to- quan ti za ti on-in-llms/
  63. References -🌸Introducing The World’s Largest Open Mul ti lingual Language

    Model: BLOOM🌸 https://bigscience.huggingface.co/blog/bloom -Sasha Luccioni https://x.com/SashaMTL -Code Carbon https://github.com/mlco2/codecarbon -Hungry for Energy, Amazon, Google and Microsoft Turn to Nuclear powerhttps:// www.ny ti mes.com/2024/10/16/business/energy-environment/amazon-google- microsoft-nuclear-energy.html -Addi ti on is All You Need for Energy-e ff i cient Language Models https://arxiv.org/ abs/2410.00907 -Energy-e ff i cient Language Models https://arxiv.org/abs/2410.00907