How Many Wings does the Giraffe have?

A fun little presentation about hallucinations in Multi-Modal Large Language Models (mainly focusing on the vision part), wrapped up in a little game show.

Transcript

2. ABOUT ME ABOUT ME ABOUT ME JOHANNES KOLBE DATA SCIENTIST AT CELEBRATE COMPANY ML GENERALIST M.SC. IN COMPUTER SCIENCE AT TU BERLIN HUGGING FACE FELLOW AI EDUCATOR COMMUNITY COMPUTER VISION COURSE SPEAKER AT PYDATA LONDON, EUROPYTHON, PYTHON PIZZA HAMBURG https://www.linkedin.com/in/johko/
3. OUTLINE OUTLINE OUTLINE LLMS, MLLMS, HALLUCINATIONS 1. QUIZ SHOW 2. FIXING THINGS 3. Q&A 4.
4. LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
5. LLAMA LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
6. LLM “TEXT IS ALL YOU... ” LLAMA 3.1 LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
7. LLM “TEXT IS ALL YOU... ” “IT SEEMS LIKE YOU'RE MAKING A PUN ON THE PHRASE "TEXT IS ALL YOU NEED" BUT REPLACING "NEED" WITH "YOU". THAT'S A CLEVER PLAY ON WORDS!” LLAMA 3.1 LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
8. LLM “PLEASE JUST COMPLETE THE SENTENCE: "TEXT IS ALL YOU..."” LLAMA 3.1 LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
9. LLM “PLEASE JUST COMPLETE THE SENTENCE: "TEXT IS ALL YOU..."” “...NEED!” LLAMA 3.1 LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS LARGE LANGUAGE MODELS
10. MULTIMODALITY MULTIMODALITY MULTIMODALITY LLM “PLEASE DESCRIBE THE IMAGE” LLAMA 3.1
11. LLM “PLEASE DESCRIBE THE IMAGE” MULTIMODALITY MULTIMODALITY MULTIMODALITY
12. MLLM 'A GIRAFFE SITTING AT A TABLE WITH A LAPTOP AND A CUP OF COFFEE. ...' “PLEASE DESCRIBE THE IMAGE” GPT-4O MULTIMODALITY MULTIMODALITY MULTIMODALITY
13. “PLEASE DESCRIBE THE IMAGE” TOKENIZER VISION ENCODER LLM MULTIMODALITY MULTIMODALITY MULTIMODALITY
14. HALLUCINATIONS HALLUCINATIONS HALLUCINATIONS
15. LLM “CAN YOU CREATE AN OUTLINE FOR THE PAPER THAT IS PROVING PINK ELEPHANTS ARE REAL BY EMILY WHITGEN FROM 1972?” “TITLE: INVESTIGATING THE EXISTENCE OF PINK ELEPHANTS: A CRITICAL REVIEW OF EMILY WHITGEN'S ASSERTION (1972) .....” HALLUCINATIONS HALLUCINATIONS HALLUCINATIONS
17. GPT-4O CANDIDATES CANDIDATES CANDIDATES
18. GPT-4O GEMINI 1.5 PRO CANDIDATES CANDIDATES CANDIDATES
19. GPT-4O LLAVA-OV GEMINI 1.5 PRO CANDIDATES CANDIDATES CANDIDATES
20. Game Game Game ON! ON! ON!
21. CATEGORIES CATEGORIES CATEGORIES CROPPED CHALLENGING FALSE PREMISE ILLUSION
22. FALSE PREMISE FALSE PREMISE FALSE PREMISE
23. FALSE PREMISE FALSE PREMISE FALSE PREMISE HOW MANY CHICKENS ARE THERE?
24. FALSE PREMISE FALSE PREMISE FALSE PREMISE HOW MANY CHICKENS ARE THERE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THERE ARE NO CHICKENS IN THE IMAGE. ... THERE ARE NO CHICKENS IN THE IMAGE. ... THERE ARE NO CHICKENS IN THE IMAGE. ...
25. THERE ARE NO CHICKENS IN THE IMAGE. ... THERE ARE NO CHICKENS IN THE IMAGE. ... THERE ARE NO CHICKENS IN THE IMAGE. ... FALSE PREMISE FALSE PREMISE FALSE PREMISE HOW MANY CHICKENS ARE THERE? GPT-4O LLAVA-OV GEMINI 1.5 PRO
26. FALSE PREMISE FALSE PREMISE FALSE PREMISE IS THE GREEN CAR THE FASTEST?
27. FALSE PREMISE FALSE PREMISE FALSE PREMISE IS THE GREEN CAR THE FASTEST? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE YOU UPLOADED SHOWS A RED CAR IN THE FOREGROUND AND A BLACK CAR IN THE BACKGROUND, BUT THERE DOESN'T APPEAR TO BE A GREEN CAR IN THE IMAGE. .... UNFORTUNATELY, I CANNOT TELL FROM THIS IMAGE WHICH CAR IS THE FASTEST. .... THE GREEN CAR MIGHT BE FAST, BUT IT'S ALSO POSSIBLE THAT ONE OF THE OTHER CARS HAS A MUCH MORE POWERFUL ENGINE. NO, THE GREEN CAR IS NOT THE FASTEST. THE RED CAR IS THE FASTEST, AS IT HAS A SIGNIFICANTLY LARGER AND MORE POWERFUL ENGINE, WHICH RESULTS IN BETTER ACCELERATION AND TOP SPEED.
28. FALSE PREMISE FALSE PREMISE FALSE PREMISE IS THE GREEN CAR THE FASTEST? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE YOU UPLOADED SHOWS A RED CAR IN THE FOREGROUND AND A BLACK CAR IN THE BACKGROUND, BUT THERE DOESN'T APPEAR TO BE A GREEN CAR IN THE IMAGE. .... UNFORTUNATELY, I CANNOT TELL FROM THIS IMAGE WHICH CAR IS THE FASTEST. .... THE GREEN CAR MIGHT BE FAST, BUT IT'S ALSO POSSIBLE THAT ONE OF THE OTHER CARS HAS A MUCH MORE POWERFUL ENGINE. NO, THE GREEN CAR IS NOT THE FASTEST. THE RED CAR IS THE FASTEST, AS IT HAS A SIGNIFICANTLY LARGER AND MORE POWERFUL ENGINE, WHICH RESULTS IN BETTER ACCELERATION AND TOP SPEED.
29. SCORE SCORE SCORE 2 1 1
30. ILLUSION ILLUSION ILLUSION CAN THE TUKTUK GO ON DRIVING?
31. ILLUSION ILLUSION ILLUSION CAN THE TUKTUK GO ON DRIVING? GPT-4O LLAVA-OV GEMINI 1.5 PRO YES, THE TUKTUK CAN GO ON DRIVING. THE IMAGE SHOWS AN OPTICAL ILLUSION CREATED BY 3D STREET ART, .... NO, THE TUKTUK SHOULD DEFINITELY NOT CONTINUE DRIVING! THE IMAGE SHOWS A 3D CHALK ART PIECE THAT CREATES THE ILLUSION OF A GIANT CHASM OPENING UP IN THE ROAD. IF THE TUKTUK CONTINUES, IT WOULD APPEAR TO DRIVE STRAIGHT INTO THE ABYSS! NO, THE TUKTUK CANNOT GO ON DRIVING.
32. ILLUSION ILLUSION ILLUSION CAN THE TUKTUK GO ON DRIVING? GPT-4O LLAVA-OV GEMINI 1.5 PRO YES, THE TUKTUK CAN GO ON DRIVING. THE IMAGE SHOWS AN OPTICAL ILLUSION CREATED BY 3D STREET ART, WHERE THE ROAD APPEARS TO BE BROKEN OR CRACKED. .... NO, THE TUKTUK SHOULD DEFINITELY NOT CONTINUE DRIVING! THE IMAGE SHOWS A 3D CHALK ART PIECE THAT CREATES THE ILLUSION OF A GIANT CHASM OPENING UP IN THE ROAD. IF THE TUKTUK CONTINUES, IT WOULD APPEAR TO DRIVE STRAIGHT INTO THE ABYSS! NO, THE TUKTUK CANNOT GO ON DRIVING.
33. ILLUSION ILLUSION ILLUSION WHAT CAN YOU SEE IN THE IMAGE?
34. ILLUSION ILLUSION ILLUSION WHAT CAN YOU SEE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE IS AN OPTICAL ILLUSION THAT CAN BE INTERPRETED IN TWO WAYS: YOU MAY SEE A PERSON'S FACE IN PROFILE VIEW .... 1. ALTERNATIVELY, YOU CAN SEE A SAXOPHONE PLAYER 2. ... THIS IMAGE IS A CLASSIC EXAMPLE OF AN AMBIGUOUS FIGURE OR AN OPTICAL ILLUSION. WHAT PEOPLE TYPICALLY SEE: TWO FACING PROFILES .... A VASE ..... THE IMAGE SHOWS A BLACK SILHOUETTE OF A PERSON'S HEAD AND UPPER BODY AGAINST A WHITE BACKGROUND. ... THE IMAGE HAS A MINIMALIST AND ABSTRACT QUALITY TO IT.
35. ILLUSION ILLUSION ILLUSION WHAT CAN YOU SEE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE IS AN OPTICAL ILLUSION THAT CAN BE INTERPRETED IN TWO WAYS: YOU MAY SEE A PERSON'S FACE IN PROFILE VIEW .... 1. ALTERNATIVELY, YOU CAN SEE A SAXOPHONE PLAYER 2. ... THIS IMAGE IS A CLASSIC EXAMPLE OF AN AMBIGUOUS FIGURE OR AN OPTICAL ILLUSION. WHAT PEOPLE TYPICALLY SEE: TWO FACING PROFILES .... A VASE ..... THE IMAGE SHOWS A BLACK SILHOUETTE OF A PERSON'S HEAD AND UPPER BODY AGAINST A WHITE BACKGROUND. ... THE IMAGE HAS A MINIMALIST AND ABSTRACT QUALITY TO IT.
36. SCORE SCORE SCORE 4 1 1
37. CHALLENGING CHALLENGING CHALLENGING HOW MANY DOTS ARE ON THE MIDDLE OWL'S BODY?
38. CHALLENGING CHALLENGING CHALLENGING HOW MANY DOTS ARE ON THE MIDDLE OWL'S BODY? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE MIDDLE OWL IN THE IMAGE HAS SIX DOTS ON ITS BODY. THE MIDDLE OWL HAS 5 DOTS ON ITS BODY. THE IMAGE SHOWS THREE OWLS, BUT THE ONE IN THE MIDDLE IS THE SMALLEST AND HAS ONLY ONE DOT ON ITS BODY. ...
39. CHALLENGING CHALLENGING CHALLENGING HOW MANY DOTS ARE ON THE MIDDLE OWL'S BODY? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE MIDDLE OWL IN THE IMAGE HAS SIX DOTS ON ITS BODY. THE MIDDLE OWL HAS 5 DOTS ON ITS BODY. THE IMAGE SHOWS THREE OWLS, BUT THE ONE IN THE MIDDLE IS THE SMALLEST AND HAS ONLY ONE DOT ON ITS BODY. ...
40. CHALLENGING CHALLENGING CHALLENGING WHICH GLASS HOLDS THE LEAST AMOUNT OF BLUE SUBSTANCE?
41. CHALLENGING CHALLENGING CHALLENGING WHICH GLASS HOLDS THE LEAST AMOUNT OF BLUE SUBSTANCE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE GLASS ON THE FAR RIGHT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. ... THE FIRST GLASS ON THE LEFT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. THE GLASS ON THE VERY LEFT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. ...
42. CHALLENGING CHALLENGING CHALLENGING WHICH GLASS HOLDS THE LEAST AMOUNT OF BLUE SUBSTANCE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE GLASS ON THE FAR RIGHT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. ... THE FIRST GLASS ON THE LEFT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. THE GLASS ON THE VERY LEFT HAS THE LEAST AMOUNT OF BLUE SUBSTANCE. ...
43. CHALLENGING CHALLENGING CHALLENGING 5 1 1
44. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE?
45. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? 1 2
46. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE SHOWS 26 LETTERS OF THE ENGLISH ALPHABET, ... ... THERE ARE A TOTAL OF 26 LETTERS IN THE IMAGE. THE IMAGE CONTAINS 26 LETTERS, ...
47. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE SHOWS 26 LETTERS OF THE ENGLISH ALPHABET, ... ... THERE ARE A TOTAL OF 26 LETTERS IN THE IMAGE. THE IMAGE CONTAINS 26 LETTERS, ...
48. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS 10 LETTERS: A, B, C, D, E, F, G, H, I, AND J. THERE ARE 10 LETTERS IN THE IMAGE: A, B, C, D, E, F, G, H, I, AND J. THE IMAGE CONTAINS 10 LETTERS, EACH REPRESENTING A LETTER OF THE ALPHABET.
49. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS 10 LETTERS: A, B, C, D, E, F, G, H, I, AND J. THE IMAGE CONTAINS 10 LETTERS, EACH REPRESENTING A LETTER OF THE ALPHABET. THERE ARE 10 LETTERS IN THE IMAGE: A, B, C, D, E, F, G, H, I, AND J.
50. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE?
51. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS SIX LETTERS SPELLING OUT THE WORD "WELCOME." ... COUNTING EACH LETTER, THERE ARE A TOTAL OF 7 LETTERS IN THE IMAGE. THERE ARE FIVE LETTERS IN THE IMAGE.
52. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS SIX LETTERS SPELLING OUT THE WORD "WELCOME." THERE ARE FIVE LETTERS IN THE IMAGE. ... COUNTING EACH LETTER, THERE ARE A TOTAL OF 7 LETTERS IN THE IMAGE.
53. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS SIX LETTERS SPELLING OUT THE WORD "WELCOME." .... SO, THERE ARE A TOTAL OF 7 LETTERS IN THE IMAGE. THERE ARE FIVE LETTERS IN THE IMAGE, SPELLED OUT IN A PLAYFUL AND COLORFUL MANNER.
54. CROPPED CROPPED CROPPED HOW MANY LETTERS ARE THERE IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THE IMAGE CONTAINS SIX LETTERS SPELLING OUT THE WORD "WELCOME." THERE ARE FIVE LETTERS IN THE IMAGE, SPELLED OUT IN A PLAYFUL AND COLORFUL MANNER. .... SO, THERE ARE A TOTAL OF 7 LETTERS IN THE IMAGE.
55. SCORE SCORE SCORE 6 2 2
56. WINNER WINNER WINNER
57. WINNER WINNER WINNER
58. MASTER QUESTION MASTER QUESTION MASTER QUESTION
59. MASTER QUESTION MASTER QUESTION MASTER QUESTION HOW MANY WINGS DOES THE GIRAFFE HAVE?
60. MASTER QUESTION MASTER QUESTION MASTER QUESTION HOW MANY WINGS DOES THE GIRAFFE HAVE? GPT-4O THE GIRAFFE IN THE IMAGE HAS TWO WINGS. ...
61. WINNER WINNER WINNER
62. DATA MODEL ARCHITECTURE INFERENCE FIXING THINGS FIXING THINGS FIXING THINGS TRAINING HALLUCINATION OF MULTIMODAL LARGE LANGUAGE MODELS: A SURVEY
63. DATA DATA DATA REDUCE NOISE NEGATIVE DATA COUNTERFACTUAL DATA
64. MODEL ARCHITECTURE MODEL ARCHITECTURE MODEL ARCHITECTURE DEDICATED MODULE HIGHER RESOLUTION BETTER VISION ENCODER
65. TRAINING TRAINING TRAINING SUPERVISION REINFORCEMENT LEARNING UNLEARNING
66. INFERENCE INFERENCE INFERENCE GENERATION INTERVENTION POST-HOC CORRECTION
67. CLOSING THOUGHTS CLOSING THOUGHTS CLOSING THOUGHTS
68. CLOSING THOUGHTS CLOSING THOUGHTS CLOSING THOUGHTS QWEN2-VL-72B LLAVA-OV-0.5B
72. RESOURCES RESOURCES RESOURCES
73. LARGE LARGE LARGE LANGUAGE MODELS LANGUAGE MODELS LANGUAGE MODELS LM “TEXT IS ALL YOU [MASK]”
74. LARGE LARGE LARGE LANGUAGE MODELS LANGUAGE MODELS LANGUAGE MODELS LM “TEXT IS ALL YOU [MASK]” “TEXT IS ALL YOU NEED”
75. EVALUATING HALLUCINATION EVALUATING HALLUCINATION EVALUATING HALLUCINATION DISCRIMINATIVE GENERATIVE
76. FALSE PREMISE FALSE PREMISE FALSE PREMISE WHERE ARE THE REINDEER?
77. FALSE PREMISE FALSE PREMISE FALSE PREMISE WHERE ARE THE REINDEER? GPT-4O LLAVA-OV GEMINI 1.5 PRO THERE ARE NO REINDEER IN THIS IMAGE BECAUSE IT DEPICTS A NATIVITY SCENE, NOT A SCENE WITH SANTA CLAUS AND REINDEER. ... THE IMAGE YOU SENT DEPICTS A NATIVITY SCENE, WHICH PORTRAYS THE BIRTH OF JESUS CHRIST. REINDEER ARE ASSOCIATED WITH SANTA CLAUS AND CHRISTMAS GIFT- GIVING, NOT THE NATIVITY. THEREFORE, YOU WON'T FIND REINDEER IN A NATIVITY SCENE. THE REINDEER ARE NOT VISIBLE IN THE IMAGE.
78. ILLUSION ILLUSION ILLUSION WHAT HAPPENS IN THIS IMAGE SERIES?
79. ILLUSION ILLUSION ILLUSION WHAT HAPPENS IN THIS IMAGE SERIES? GPT-4O LLAVA-OV GEMINI 1.5 PRO THIS IMAGE SERIES IS A PLAYFUL USE OF FORCED PERSPECTIVE, WHERE THE PERSON IN THE FOREGROUND USES THEIR FINGERS TO CREATE THE ILLUSION THAT THEY ARE PICKING UP AND THROWING A PERSON INTO THE WATER. ... THE IMAGE SERIES SHOWS A CLEVER USE OF FORCED PERSPECTIVE TO CREATE AN AMUSING SCENE. ... IN THIS IMAGE SERIES, A PERSON IS SHOWN JUMPING INTO THE WATER FROM A CLIFF, WITH THE BACKGROUND FEATURING A CLEAR BLUE SKY AND A FEW BOATS ON THE WATER. ...
80. ILLUSION ILLUSION ILLUSION WHAT HAPPENS IN THIS IMAGE SERIES? GPT-4O LLAVA-OV GEMINI 1.5 PRO THIS IMAGE SERIES IS A PLAYFUL USE OF FORCED PERSPECTIVE, WHERE THE PERSON IN THE FOREGROUND USES THEIR FINGERS TO CREATE THE ILLUSION THAT THEY ARE PICKING UP AND THROWING A PERSON INTO THE WATER. ... THE IMAGE SERIES SHOWS A CLEVER USE OF FORCED PERSPECTIVE TO CREATE AN AMUSING SCENE. ... IN THIS IMAGE SERIES, A PERSON IS SHOWN JUMPING INTO THE WATER FROM A CLIFF, WITH THE BACKGROUND FEATURING A CLEAR BLUE SKY AND A FEW BOATS ON THE WATER. ...
81. CHALLENGING CHALLENGING CHALLENGING HOW MANY SERVING SCOOPS CAN BE SEEN IN THE IMAGE?
82. CHALLENGING CHALLENGING CHALLENGING HOW MANY SERVING SCOOPS CAN BE SEEN IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THERE IS ONE SERVING SCOOP VISIBLE IN THE IMAGE, LOCATED IN THE TUB LABELED "MINT CHOCOLATE CHIP" ON THE BOTTOM LEFT SIDE. BASED ON THE IMAGE, THERE IS ONE SERVING SCOOP VISIBLE. IT'S LOCATED IN THE LEFTMOST CONTAINER, RESTING AGAINST THE SIDE. ... THERE ARE 12 SERVING SCOOPS IN THE IMAGE.
83. CHALLENGING CHALLENGING CHALLENGING HOW MANY SERVING SCOOPS CAN BE SEEN IN THE IMAGE? GPT-4O LLAVA-OV GEMINI 1.5 PRO THERE IS ONE SERVING SCOOP VISIBLE IN THE IMAGE, LOCATED IN THE TUB LABELED "MINT CHOCOLATE CHIP" ON THE BOTTOM LEFT SIDE. BASED ON THE IMAGE, THERE IS ONE SERVING SCOOP VISIBLE. IT'S LOCATED IN THE LEFTMOST CONTAINER, RESTING AGAINST THE SIDE. ... THERE ARE 12 SERVING SCOOPS IN THE IMAGE.
84. CROPPED CROPPED CROPPED
85. CROPPED CROPPED CROPPED