Appearances Matter: AI Learning from Text and Images

Is a picture really worth a thousand words? Recent progress in AI has in fact been driven by learning from both images and text, opening up exciting new possibilities. Computers are now increasingly able to extract insights from vast quantities of image and text data. This opens up new opportunities, but also leads to notable risks that need to be considered. Can a thoughtful approach towards AI allow us to obtain systems that seem a little more human and better guide us in our everyday life?

Transcript

1. Appearances Matter: AI Learning from Text and Images Gerard de Melo http://gerard.demelo.org Appearances Matter: AI Learning from Text and Images Gerard de Melo http://gerard.demelo.org https://commons.wikimedia.org/wiki/File:Muliple_colored_pencils_07.jpg
2. https://zh.wikipedia.org/zh-hk/ 柏林
3. Image: Diana Parkhouse
4. Image: Diana Parkhouse
5. Learning from Examples Learning from Examples Learning from Examples Learning from Examples cat
6. Learning from Examples Learning from Examples Learning from Examples Learning from Examples KI lernt anhand von Beispielen AI learns from examples.
7. Learning from Raw Data Learning from Raw Data Learning from Raw Data Learning from Raw Data Image: Olga Andreyanova
8. Learning from Raw Data Learning from Raw Data Learning from Raw Data Learning from Raw Data An Example Pair to Learn from Image: Olga Andreyanova
9. Learning from Raw Data Learning from Raw Data Learning from Raw Data Learning from Raw Data Berlin ist eine lebendige Stadt.
10. Learning from Raw Data Learning from Raw Data Learning from Raw Data Learning from Raw Data Berlin ist eine lebendige Stadt. An Example to Learn from
11. Image: https://500px.com/photo/96738645/street-bookstore-by-pablo-tarrero Learning from many millions of examples
12. Mouse brain microscopy, adapted from Na Ji, UC Berkeley Neural Connections in the Brain
13. Learning Models Learning Models Learning Models Learning Models
14. Connecting Vision and Language Connecting Vision and Language Connecting Vision and Language Connecting Vision and Language x x x x A cat sitting on a mat looking out the door https://commons.wikimedia.org/wiki/File:Nebelung_cat_looking_out_door.jpg / https://commons.wikimedia.org/wiki/File:Dog_costume_parade_in_Sunset_Park_%2816432%29.jpg A dog wearing a USPS mail carrier costume
15. Generation Models Generation Models Generation Models Generation Models
16. Generation Example Generation Example Generation Example Generation Example
17. Generation by Generation by Combining Vision and Language Combining Vision and Language Generation by Generation by Combining Vision and Language Combining Vision and Language Teddy bears working on new AI research underwater with 1990s technology https://commons.wikimedia.org/wiki/File:DALL-E_2_artificial_intelligence_digital_image_generated_photo.jpg
18. Challenge: Challenge: Responsible AI Responsible AI Challenge: Challenge: Responsible AI Responsible AI Responsible Use of AI Image:Kev Costello
19. Human Understanding Human Understanding Human Understanding Human Understanding DeeepMind’s TAP-Vid
20. Learning from Videos Learning from Videos Learning from Videos Learning from Videos Geng et al. Character Matters: Video Story Understanding with Character-Aware Relations
21. Masked-Piper. Babajide Owoyele, James Trujillo, Gerard de Melo, Wim Pouw (2022)
22. Understanding Human Perception Understanding Human Perception Understanding Human Perception Understanding Human Perception Tugba Kulahcioglu, Gerard de Melo. Predicting Semantic Signatures of Fonts
23. Understanding Human Perception Understanding Human Perception Understanding Human Perception Understanding Human Perception Van Rompay and Pruyn (2011) + de Sousa et al. (2020). Journal of Sensory Studies Choice of font affects brand credibility, price expectations, even assumed acidity, sweetness
24. Understanding Human Perception Understanding Human Perception Understanding Human Perception Understanding Human Perception Tugba Kulahcioglu, Gerard de Melo. FontLex: A Typographical Lexicon based on Affective Associations / Tugba Kulahcioglu, Gerard de Melo. Paralinguistic Recommendations for Affective Word Clouds. Proc. IUI The Smurfs Scream
25. Understanding Human Perception Understanding Human Perception Understanding Human Perception Understanding Human Perception Contentment Fear Dobler et al. Art Creation with Multi-Conditional StyleGANs
26. Towards Trustworthy AI: Towards Trustworthy AI: Medical Applications Medical Applications Towards Trustworthy AI: Towards Trustworthy AI: Medical Applications Medical Applications
27. Towards Trustworthy AI: Towards Trustworthy AI: Daily Life Applications Daily Life Applications Towards Trustworthy AI: Towards Trustworthy AI: Daily Life Applications Daily Life Applications Where’s the nearest exit?
28. Image: Adapted from https://techcrunch.com/2017/12/22/fin-assistant/ Contact: http://gerard.demelo.org linkedin.com/in/gdemelo