Running llama.cpp on the CPU

A 5 minute lightning talk introducing llama.cpp, showing how we can run gguf models on the CPU without needing a GPU. I show llama2, WizardCoder and Llava multimodal, with command line arguments and links to the source gguf files.
To be written up on: https://notanumber.email/ and https://ianozsvald.com/
License - Creative Commons By Attribution

Transcript

1. llama.cpp – what do we get? PyDataLondon 2024-01 lightning talk @IanOzsvald – ianozsvald.com
2. No need for a GPU+VRAM Llama.cpp runs on CPU+RAM Nothing sent off your machine llama.cpp By [ian]@ianozsvald[.com] Ian Ozsvald X
3. Prototype ideas! By [ian]@ianozsvald[.com] Ian Ozsvald llama-2-7b-chat.Q5_K_M.gguf 5GB on disk and in RAM, near real time
4. Experiment with coding assistants (base llama2 model not good at this) By [ian]@ianozsvald[.com] Ian Ozsvald
5. WizardCoder is good (tuned llama2) By [ian]@ianozsvald[.com] Ian Ozsvald wizardcoder-python-34b -v1.0.Q5_K_S.gguf 22GB on disk & RAM 15s for example You can replace CoPilot with this for completions
6. Llava multi-modal Extract facts from images? By [ian]@ianozsvald[.com] Ian Ozsvald llava-v1.5-7b-Q4_K.gguf 4GB on disk & RAM 5s for example llama.cpp provides ./server
7. By [ian]@ianozsvald[.com] Ian Ozsvald
8. Try Mixtral, Phi2, UniNER etc Wild wild west (Aug+ is sane) What could you prototype? Let’s discuss in the break – what are you building? Summary By [ian]@ianozsvald[.com] Ian Ozsvald