Do not LLM before you can walk

This presentation, while providing some historical insights into Natural Language Processing (NLP) advancements, highlights the risks of hastily jumping onto the AI bandwagon without a well-defined strategy. We explore the possible challenges in neglecting key factors such as data quality, monitoring and domain expertise while building data products. By understanding the past, attendees will gain a sharper perspective on building reliable and impactful AI data products. Let’s walk confidently before sprinting into the AI revolution, ensuring a solid foundation for success.

Transcript

1. Do not LLM before you can walk AI strategy in light of NLP breakthroughs Massimo Belloni - London, UK - December 2023
2. Massimo Belloni Data Science Manager @ Bumble Inc. • Machine Learning Engineer (󰏢) - lived and worked in Rotterdam (󰐗), London (󰏅) and Barcelona (󰎼) • Currently Data Science Manager at Bumble Inc. for Integrity & Safety and MLOps @massibelloni /massibelloni massibelloni@gmail.com
3. 87%
4. 87% Data Science projects never make it to production Venture Beat, 2019
5. ~1980 Convolutional Neural Networks 1996 LSTMs 2017 “Attention is All You Need” Transformer 1998 LeNet 2019 GPT-2 1989 Backpropagation in real world 2012 AlexNet on ImageNet 2021 DALL-E 2022 StableDiffusion 2014 GANs 30 Nov 2022 ChatGPT March 2022 GPT 3.5
6. 0 ChatGPT ML didn’t exist
7. Technology Product
8. Technology Product ChatGPT RecSys Dynamic Pricing Sentiment Analyser Logistic Regression Gradient Boosted Tree Convolutional Neural Networks BERT (etc). Stable Diffusion GPT 3 CLIP
9. Product • 👥 Directly used by wide range of people through interfaces and user experience • 🎯 Solves specific problems with quantifiable outcomes • 🎨 Multi-faceted iteration cycles, multi-disciplinary and cross-functional
10. Technology • 🚀 Unlock capabilities - makes ideas or processes feasible and remunerative • 🧠 Requires specific and vertical knowledge to be understood, built and improved • ⌛ Usually long iteration cycles and strategic investments in Research & Development
11. Technology Product
12. Technology Product A Large Language Model able to understand and predict long and complex sentences one word at a time, creating coherent and comprehensive paragraphs with human-level quality. A complete and user friendly human-level chatbot experience, with ad-hoc integrations and the possibility to call APIs and URLs over the internet
13. Technology Product
14. The recipe for Failure • 🤔 Unclear goals and objectives Are we actually improving an existing business process in a measurable and quantifiable way? • 🏗 Lack of engineering skills How far in the ML lifecycle are we able to go with internal resources? Are we able to communicate with other systems? • 📖 Poor experiments’ management Are all the experiments tracked and replicable? Can we trace back the history (datasets, code, parameters) of all the artifacts we are generating?
15. Process understanding and model’s design Deployment infrastructure and first PoC Model improvement and experimentation strategy
16. What are we solving for?
17. Technology Product ChatGPT RecSys Dynamic Pricing Sentiment Analyser Logistic Regression Gradient Boosted Tree Convolutional Neural Networks BERT (etc). Stable Diffusion GPT 3 CLIP
18. Don’t fall for the LLM trap • 🧱 LLMs are a ML technology, not a final product Luckily or not, even after GenAI boom the rules of the game didn’t change. • 📚 Wide offering of different LLMs with different licensing The OSS community is catching up fast with proprietary solutions and the market is rich of free alternatives (GPUs allowing!) • 💸 Huge interest, opportunities and investments Post-2022 LLMs are opening up incredible opportunities and human-level performance in text understanding and generation - what was science fiction 2 years ago is now a product idea worth exploring!
19. Thanks!