Finding the needle: a deep dive into the rewriting of Haystack

Haystack is an open-source framework. With Haystack you can compose various NLP tools to build applications, with a particular focus on Large Language Models. Haystack was built before the “ChatGPT revolution”. Same as many others in this industry, we had to question all the existing assumptions in order to adapt, and we had to do it fast.

In this talk, we'll explore the motivations behind the refactoring, the challenges we faced, and the outcomes achieved through this intensive process. From rethinking many of the original abstractions, all the way up to growing a vibrant community of users and contributors, we’ll share the key strategies and techniques employed during this journey.

Whether you're a seasoned open-source contributor or a curious enthusiast, this talk promises to uncover valuable insights and lessons learned from the evolution of Haystack.

Transcript

1. Finding the Needle a deep dive into the rewriting of Haystack PyCon USA - 2024
2. @TUANACELIK /IN/TUANACELIK Tuana Çelik Lead Developer Advocate TUANACELIK Massimiliano Pippi Open Source Engineering Lead /IN/MASCI/ MASCI
3. This is a story of a framework redesign
4. What is Haystack? Haystack is an open-source framework for building production-ready AI applications It lets you quickly try out the latest AI models while being flexible and easy to use Think chatbots, document search, question-answering TL;DR: Anything that makes use of modern language models and AI techniques..
5. 2020 First release
6. 2020 First release
7. 2022 Things changed....significantly
8. Given the documents, answer the question. Documents: {{ documents}} Question: {{ question }}
9. Given the documents, answer the question. Documents: {{ documents}} Question: {{ question }}
10. Given the documents, answer the question. Documents: {{ documents}} Question: {{ question }}
11. Question Answering Given the documents, answer the question. Documents: {{ documents}} Question: {{ question }} Summarization Summarize the following text. Text: {{ text}} Question Generation Given the following document, generate some questions Document: {{ document}} Question:
12. Retrieval Genration Query embedding Prompt Response
13. Retrieval Genration Query embedding Prompt Response Reranking
14. Component Component Component Pipeline
15. Text Embeddings Retrieval Vector Databases Prompting & LLMs Maybe some classifiers.. Maybe reranking..? .....and on and on and on.. To Build AI Applications
16. Haystack was first officially released in 2020 (read: before the LLM boom) Semantic search, extractive QA, retrieval, even preparing and writing documents into a vector database Components (nodes) and pipelines It allowed users to combine their desired language models with their data source 👍 The pipeline-component structure is a great abstraction for building composable LLM application 👎 There were some assumptions that were made, as well as some design mishaps A little bit of History
17. Haystack 1.x 1. 2. 3. 4. We lived, we learned, we evolved 💪 Special-General Mixture Overexposure Information Leakage Conjoined Components
18. Overexposure ignorance is bliss 🤔 🙂 😳 😡
19. Special-General Mixture this craziness, it's too much. 😵‍💫
20. Information Leakage some people just want to watch the world burn 🤔 🙂
21. Information Leakage some people just want to watch the world burn 😭 🧐
22. Conjoined Components you had one job 😮 😌 😦 😰
23. Pipeline Structure
24. Pipeline Structure 🙂 😊
25. Pipeline Structure 🤔 🤬 🤦‍♀️
26. Component Component Component Pipeline Haystack 2.0 Pipelines & Components ✅ A component is responsible of ONE. THING. ONLY. Descriptive components UNIFORM INTERFACE Pipelines that are able to branch and cycle 📚 Announced March 11, 2024
27. A simple interface for the creation of components. User defined number of inputs and outputs. Components 📚 Announced March 11, 2024
28. A simple interface for the creation of components. User defined number of inputs and outputs. Components Query documents 📚 Announced March 11, 2024
29. Documentation Bliss Before: PromptNode (a God among nodes)
30. Documentation Bliss After: A simple generator, doing honest work
31. Pipelines Input 1 Input 2 Output 1 Output 2
32. Pipelines Input 1 Input 2 Output 1 Output 2
33. Pipelines Input 1 Input 2 Output 1 Output 2 Input 2 Input 1 Output 1 Output 2
34. Pipelines Component Component Component Pipeline
35. Pipelines Component Component Pipeline
36. Pipelines Pipeline
37. Pipelines Pipeline
38. Document Stores Pipeline
39. ConditonalRouter WebSearch Generator PromptBuilder And..? go_to_websearch answer Answer ... 📚 Tutorial
40. And..? Embedder Embedding Retriever BM25Retriever DocumentJoiner Ranker Query 📚 Tutorial
41. PromptBuilder Generator PromptBuilder Generator OutputValidator And..? Invalid replies & Error Message Valid Replies Unstructured Text 📚 Tutorial
42. And..? Generator PromptBuilder Embedder Query Retriever Answer Prompt Response Query embedding Query Query embedding
43. Inspo
44. Prototype Self-hosted app Production ready GOTO Prod
45. Pipelines as middleware
46. Observe
47. Integrations
48. Build like a Pro Release notes and breaking change policiy
49. Deployment guide
50. pip install haystack-ai
51. 📍 Ballroom A Everything is a graph, including LLM Applications (and that’s handy) 📆 May 18th, 1:30-2:15PM @TUANACELIK /IN/TUANACELIK Tuana Çelik TUANACELIK
52. Exhibit Hall A, Booth 330 📍Find us at 🥂 AI Happy Hour with DataStax 📍The Standard Market and Pint House 📆 May17th, 6PM