Leveraging Generative AI to help the blind and visually impaired people

Transcript

1. Be My Eyes... Leveraging Generative AI to help the blind and visually impaired people
2. © 2024 Redpanda Data About the presenter 2 Dunith Dhanushka Senior Developer Advocate, Redpanda Data • Event streaming, real-time analytics, and stream processing enthusiast • Frequent blogger, speaker, and an educator
3. © 2024 Redpanda Data 3 The Story
4. © 2024 Redpanda Data Trying to solve an old problem… 4
5. © 2024 Redpanda Data 5 Then Came Generative AI!
6. © 2024 Redpanda Data 6 Generative Models with Vision Capabilities • Open AI - Open AI • Llama 3 - Meta • Claude 3 - Anthropic • Vertex AI - Google • Mistral 7B, Pixtral 12B - Mistral
7. © 2023 REDPANDA DATA 7
8. © 2023 REDPANDA DATA Vision capabilities in action… 8
9. 9 Smart personal Assistant for blind people
10. © 2024 Redpanda Data 10 Design Goals
11. © 2024 Redpanda Data 11 Leverage their hearing ability The blind can listen really well…
12. © 2024 Redpanda Data 12 Make it practical Make it wearable and add push buttons
13. © 2024 Redpanda Data 13 Make it affordable Make it cost-efficient and economical in long term
14. © 2024 Redpanda Data 14 How it works?
15. © 2024 Redpanda Data 15 1. User presses the push button 2. Camera takes a photo 3. Image is sent to the LLM 4. Image description is generated 5. Image is narrated to the user
16. © 2024 Redpanda Data Implementation Options 16 Building a Smart Device with Raspberry Pi Building a smartphone app
17. © 2024 Redpanda Data With Raspberry Pi Affordable, flexible, but complex 17
18. © 2024 Redpanda Data Foundation - Raspberry Pi 5 How the device does the processing? 18 Affordable compute for reasonable performance
19. © 2024 Redpanda Data 19 Adding necessary modules
20. © 2024 Redpanda Data Python Program 1. Takes the image captured by the camera. 2. Sends the base64 encoded image to LLM over the Internet. 3. Receives the response and converts it to audio via a text-to-speech converter. 20
21. © 2024 Redpanda Data 21
22. © 2024 Redpanda Data 22 Code walkthrough
23. © 2024 Redpanda Data 23 An alternative with a smartphone
24. © 2024 Redpanda Data Why a Smartphone? 24 Camera Storage Connectivity Audio Power Performance
25. © 2024 Redpanda Data Activation and Interactivity with Voice Commands Use a voice assistant like Siri 25
26. © 2024 Redpanda Data Raspberry Pi FTW! • Affordable • Flexible • Offers reasonable performance 26
27. © 2024 Redpanda Data 27 Making it more affordable
28. © 2024 Redpanda Data Using Open-source Local LLMs 28 Run LLMs on Ollama
29. © 2024 Redpanda Data Working with Local LLMs Pros • Reduced latency • Itʼs free! • Data privacy Cons • Performance Models with large parameters requires more storage and processing power. GPUs are preferred over CPUs) 29
30. © 2024 Redpanda Data 30 Making it scale-out Build it for the masses
31. © 2024 Redpanda Data 31 Integrated Monitoring Solution
33. © 2024 Redpanda Data Potential use cases… • Remote patient monitoring (ideal for care homes and hospitals) • Monitoring the workers working in low visibility environments E.g Monitoring the vitals of fire fighters and assist them with their vision) • Personal assistant to the elderly • AI-assisted bodycam for rescue workers 33 What else we can do with this?
34. © 2024 Redpanda Data Wrap up • Blind people need a practical and affordable solution to make their lives easier. Especially, increase their confidence in mobility and social inclusion. • Image recognition and classification is an already solved problem. But using Generative AI to describe images makes it more accessible, affordable and scalable. • Redpanda and its ecosystem can take individual devices to the next level by implementing a monitoring and assistive technology for the visually impaired people. 34
35. © 2023 REDPANDA DATA Redpanda University Free, self-paced online learning https://university.redpanda.com • Learn the fundamentals of data streaming and Redpanda • Install Redpanda and use the rpk CLI to configure it • Create producers and consumers in Java, Python and NodeJS • Sign up today for free! 35
36. © 2024 Redpanda Data Thanks for joining! Letʼs keep in touch @redpandadata redpanda-data redpanda-data hello@redpanda.com