Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Finetuning LLMs on consumer GPUs
Search
Aniket Maurya
November 07, 2023
Programming
0
96
Finetuning LLMs on consumer GPUs
Aniket Maurya
November 07, 2023
Tweet
Share
More Decks by Aniket Maurya
See All by Aniket Maurya
Building RAG powered applications - PyData London 2nd April
aniketmaurya
0
26
Contributing to Lightning AI OSS
aniketmaurya
0
16
Other Decks in Programming
See All in Programming
GitLab CI/CD で C#/WPFアプリケーションのテストとインストーラーのビルド・デプロイを自動化する
hacarus
0
610
Docker_OSS_ホスティング入門
satokoki645
0
140
Native Federation: The Future of Micro Frontends in Angular
manfredsteyer
PRO
0
170
GitHub Actionsの痒いところを埋めるサードパーティーランナー
dora1998
1
250
Long journey of Ruby standard library RubyKaigi 2024
andpad
2
190
RubyGems on ruby.wasm
kateinoigakukun
0
110
RaaP
ksss
0
150
一文字エイリアスのすすめ
fujimura
0
190
Webアプリをできるだけコードを手書きしないで作ってみる
tomokusaba
2
230
[RubyKaigi 2024] Ruby Mixology 101: adding shots of PHP, Elixir, and more
palkan
0
100
Node.js v22 で変わること
yosuke_furukawa
PRO
12
4.2k
JavaScript Closure
asoluka
0
2k
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
74
8.3k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
14
8.4k
Navigating Team Friction
lara
179
13k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Building an army of robots
kneath
300
41k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.7k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
275
13k
Adopting Sorbet at Scale
ufuk
69
8.6k
The Invisible Customer
myddelton
114
12k
Documentation Writing (for coders)
carmenintech
60
4k
Bootstrapping a Software Product
garrettdimon
PRO
302
110k
Transcript
November 2023 1. Finetuning LLMs on consumer GPUs 2. LLM
Evaluation framework and datasets 3. Deep Dive into Transformers 4. Effortlessly analyze multifaceted financial documents with LlamaIndex
Finetuning LLMs on custom datasets Aniket Maurya, Developer Advocate at
Lightning AI November 2023 X.com/aniketmaurya linkedin.com/in/aniketmaurya
• Overview of LLMs • Parameter efficient finetuning with instruction
dataset • Training on consumer GPUs Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 3 Agenda
What are LLMs Lightning AI ©2023 Proprietary and Confidential. All
Rights Reserved. 4
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 5
What are LLMs
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 6
What are LLMs Source: Attention is All you Need
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 7
What are LLMs
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 8
What are LLMs
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 9
What are LLMs *Decoder
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 10
Parameter Efficient Finetuning Source : https://lightning.ai/pages/community/tutorial/lora-llm/
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 11
Parameter Efficient Finetuning
• Remove untruthfulness and toxicity • Customize the output and
tone of language • Privacy and control Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 12 Why Finetune LLMs
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 13
Finetuning LLMs on instruction dataset
• Setup model • Prepare data • Finetune the model
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 14 Finetuning LLMs
• Setup model • Prepare data • Finetune the model
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 15 Finetuning LLMs
• Setup model • Prepare data • Finetune the model
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 16 Finetuning LLMs
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 17
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 18
• 4-bit quantized finetuning and inference • Minimal code, easy
to debug and hack • TPU support • Flash-Attention 2 Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 19 Lit-GPT
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 20
Finetuning Llama on instruction dataset
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 21
Setup Model
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 22
Setup Model
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 23
Prepare Dataset
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 24
Finetune
CUDA Out Of Memory Lightning AI ©2023 Proprietary and Confidential.
All Rights Reserved. 25
• Llama 7B, fp32: ~28GB • Llama 7B, fp16: ~14GB
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 26 Memory Required to load Llama
• Activation memory • Gradient memory • Optimizer memory •
Model memory Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 27 Memory Usage
• Activation memory • Gradient memory • Optimizer memory •
Model memory Source: https://tinkerd.net/blog/machine-learning/distributed-training/ 28 Memory Usage
29 • Reduce the micro batch size Avoid OOM
30 • Reduce the model's context length • Reduce the
micro batch size Avoid OOM
31 • Reduce the model's context length • Use lower
precision • Reduce the micro batch size Avoid OOM
• 4-bit quantization 32 • Reduce the model's context length
• Use lower precision • Reduce the micro batch size Avoid OOM
72% memory reduction Lightning AI ©2023 Proprietary and Confidential. All
Rights Reserved. 33
34 • Reduce the model's context length • Use lower
precision • 4-bit quantization • Do sharding across multiple GPUs • Reduce the micro batch size Avoid OOM
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 35
Avoid OOM
• Lit-GPT with LoRA finetuning • Lower Precision and 4-bit
quantization • Distributed training and activation checkpointing Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 36 Conclusion
Lightning AI ©2023 Proprietary and Confidential. All Rights Reserved. 37
Aniket Maurya