What is multimodality, and can it generate captions for my Instagram photos?

2024 February Prague Python Pizza
https://prague.python.pizza/

Transcript

1. Multimodality Whatis AND CAN IT GENERATE CAPTIONS FOR MY INSTAGRAM PHOTOS?
2. About me • 🥑 Developer Relations Engineer at deepset • 🏗 Open source LLM Framework: Haystack • 📍 Istanbul, Turkey • 💃 Latin music/dances Bilge Yücel
3. IMAGE-TO-TEXT TODAY'S agenda 01 02 03 04 TEXT-TO-IMAGE MULTIMODALITY TEXT-TO-TEXT 05 CAPTION GENERATOR
4. text-to-text
5. text-to-text Text generation • GPT-3.5-turbo • GPT-4,Gemini-Pro, Claude, Command, Mistral…
6. multimodality
7. Multimodality refers to the capacity of a machine learning model to process and understand information from various modalities, such as text, images, videos, and audio. This capability allows the model to perform tasks that require integration of different types of data, such as creating video from a prompt or generating a text based on the image. Multimodality
8. text-to-image
9. A street cat playing a shrill pipe and drinking raki by the Bosphorus, photorealistic, natural light text-to-image Image generation • Gemini-Pro-Vision, Midjourney, Dall-E, Stable Diffusion…
10. image-to-text
11. image-to-text CAPTION GENERATION Bustling city square with street performers in the shadow of towering Gothic spires under a blue sky with wispy clouds. • OCR, image classification • GPT-4V, Gemini-Pro-Vision, CLIP…
12. Instagram Caption Generator
14. 🐳💦✨Witnessing the majestic dance of a whale, leaping high above the cerulean sea. 🌊 #WhaleWatching #NatureLovers #Ocean
16. Captionate 📸 • Generates Instagrammable captions • image-to-text + text-to-text • Haystack + Gradio
17. Haystack • Fully open-source framework built in Python for custom LLM applications
18. Haystack • Fully open-source framework built in Python for custom LLM applications • Provides tools that developers need to build state-of-the-art NLP systems
19. Haystack • Fully open-source framework built in Python for custom LLM applications • Provides tools that developers need to build state-of-the-art NLP systems • Building blocks: Pipelines & Components
20. Caption Generation Pipeline IMAGE-TO-TEXT TEXT-TO-TEXT
21. Caption Generation Pipeline IMAGE-TO-TEXT TEXT-TO-TEXT
22. Caption Generation Pipeline IMAGE-TO-TEXT TEXT-TO-TEXT
23. Caption Generation Pipeline IMAGE-TO-TEXT TEXT-TO-TEXT
24. Caption Generation Pipeline IMAGE-TO-TEXT TEXT-TO-TEXT
25. Caption Generation Pipeline
26. Caption Generation Pipeline
27. Caption Generation UI
28. Generative Question Answering
29. A wrap That's THANK YOU
30. Resources Captionate 📸 @bilgeycl Bilge Yücel Haystack @bilgeyucel Presentation