Generative AI

An introduction to how Generative AI works. A slide deck I presented at a symposium for IT staff from the Dutch defence ministry.

1. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Generative AI Vereniging Informatici Defensie - October 5, 2023 Midjourney - "a symposium for ministry of defense staff, with drones in attendance, sci-fi movie style" Ivo Jansch ivo@egeniq.com @ijansch
2. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 DISCLAIMER By the time we finish this talk it's probably already outdated Midjourney - "people attending a presentation in an auditorium. There's a time machine phonebooth from Dr Who in the corner"
3. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Chapter 1: How does Generative AI work? Midjourney - "a brain of a robot, full of neurons. Science fiction style"
4. 2005 - 2015: Rise of Deep Learning Introducing layers into the field of machine learning: pixels grouped pixels edges green hues … a cat a cat staring a cat staring at you a cat at a waterhole in the jungle
5. 2017: Invention of the Transformer Architecture Has nothing to do with The Transformers Has everything to do with a new, scalable architecture that created the ability to process large quantities of data and most importantly: - Self supervised learning - Self attention
6. Understanding attention The cat drank from the waterhole until it was full The cat drank from the waterhole until it was empty
7. Understanding attention The cat drank from the waterhole until it was full The cat drank from the waterhole until it was empty
8. 2018: GPT Generative Predict the next word(s) from a sequence of words Pre-trained Trained on a large corpus of text Transformer Built on the transformer architecture
9. Training the model Training Data 17 Gb for GPT-3 45 Gb for GPT-4 Language Model 175 billion parameters for GPT-3 1 trillion parameters for GPT-4
10. Training the model - word embeddings Cat Animal 0.3 Domestic 0.2 Hairy 0.25 Smelly 0.25 Food 0.3 PC 0.98 Mouse 0.3 other words encountered distance to those words
11. Training the model - word embeddings Cat Animal 0.3 Domestic 0.2 Hairy 0.25 Smelly 0.25 Food 0.3 PC 0.98 Mouse 0.3 Mouse Animal 0.3 Domestic 0.5 Hairy 0.25 Smelly 0.25 Food 0.7 PC 0.2 Cat 0.3 Dog Animal 0.3 Domestic 0.2 Hairy 0.25 Smelly 0.4 Food 0.25 PC 0.99 Mouse 0.7 PC Animal 0.98 Domestic 0.99 Hairy 0.7 Smelly 0.8 Food 0.8 Cat 0.98 Mouse 0.3
12. Word embeddings are useful for self-attention The cat drank from the waterhole until it was full Cat Full
13. The model 'knows' relationships between words Cat Drink Pet Water Fat Rat Dog PC Mouse Jungle Rhyme Vast
14. The model 'knows' relationships between words Cat Drink Pet Water Fat Rat Dog PC Mouse Jungle Rhyme Vast These are the 'parameters'
15. Then it starts predicting Given a prompt, what is the most likely word that comes next What is a cat? 96% 20% 94% Fat A pet An animal
16. Like humans, not always the same What is a cat? 96% 20% 94% Fat A pet An animal 'Temperature' introduces some randomness between similar answers
17. Word by word Given a prompt, what is the most likely word that comes next Input: Why does a cat drink water? Prediction: A cat
18. Word by word by word Given a prompt, what is the most likely word that comes next Input: Why does a cat drink water? A cat Prediction: needs
19. Word by word by word by word Given a prompt, what is the most likely word that comes next Input: Why does a cat drink water? A cat needs Prediction: nutrition
20. Word by word by word by word Given a prompt, what is the most likely word that comes next Done: Why does a cat drink water? A cat needs nutrition
21. Note: a model works with tokens, not words The cat drank from the waterhole until it was full All tokens are converted to numbers. 17 1345 98 45 17 2624 213 21 78 723 A language model is essentially an abstract model of relationships between numbers.
22. GPT's are "stochastic parrots" - They "understand" - Yet, they don't truly understand what they are talking about - They predict which numbers are most likely to follow other numbers, given the context of other numbers.
23. Disclaimer That was by no means a scientifically correct explanation. But explains in a very simplified way how it works.
24. If it works for words, it can also work for other things Music… Images… Film…
25. News this week:
26. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Chapter 2: Generative AI as assistant Midjourney - "a development team sitting around a table, everyone wearing AR glasses. Science fiction style"
27. Increase Coding Productivity
28. Coaching Junior Developers
29. Coaching Junior Developers
30. Designing Data Models LLMs can help with creating data models and normalization.
32. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Chapter 3: Generative AI application development Midjourney - "smart ai robots in a factory, performing tedious manual tasks for humans, science fiction style"
33. Application areas Enhancing Copy writing Gap filling Summarising Extracting Analysing Translating Converting Copy editing Models are good at: But not so suitable for factfinding / data accuracy
34. Application Approach 1: Train your own model Training Huge Dataset Custom LLM Prompt engineering Input Output
35. Application Approach 2: Model finetuning Training Additional Dataset Custom LLM Existing LLM Prompt engineering Input Output
36. Application Approach 3: Vector databases Vector Database Existing LLM Prompt engineering Additional Dataset Create Embeddings Input Output
37. Application Approach 4: LLMs as tool in a chain SQL Query Additional Dataset Factual Results Summarize Existing LLM Translate to query Input Output
38. Example usecase "Can AI assist with answering parliamentary questions?"
39. Example use case "Can AI assist with answering parliamentary questions?"
40. Application Application Proof of concept: LLM: Predict Parliamentary Questions Historic Parliamentary Questions Vector Database Create Embeddings News Prompt Engineering Formulate Response LLM: Verify against party program / prior statements LLM: Assist copy-writing for chamber, press and public Verify 💻 Party Programs Vector DB
41. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Chapter 4: The dark side Midjourney - "a postapocalyptic city run by AI, full of drones. Humans have disappeared or are enslaved. Photorealistic image"
42. Hallucinations
43. Hallucinations
44. Hallucinations
45. Hallucinations
46. Hallucinations
47. What probably happened Tokens 5+ real methods starting with SecKeyCopy
48. Copyright concerns
49. Copyright concerns The output can contain copyrighted material!
50. Security concerns Your input can end up in training data!
51. Security & Quality concerns Paper: https://arxiv.org/abs/2211.03622
52. Environmental concerns
53. Environmental concerns Source: https://blogs.nvidia.com/blog/2022/03/25/what-is-a-transformer-model
54. Existential concerns Asimov's first law of robotics: "a robot shall not harm a human, or by inaction allow a human to come to harm"
55. Existential concerns FAKE NEWS Asimov's first law of robotics: "a robot shall not harm a human, or by inaction allow a human to come to harm"
56. Things to pay attention to AI tools are useful, but: • Ensure human supervision -> Code reviews • Be transparent about the use of AI • Pay attention to Terms & Conditions of the tools you use • Keep an eye on copyright legislation • Choose the right tool for the job ◦ ChatGPT is not the only player in town ◦ Consider open source alternatives such as LLAMA ▪ https://github.com/eugeneyan/open-llms
57. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Chapter 5: The future Midjourney - "A robot and a human holding hands, watching the sunset in the distance. Science fiction style"
58. Will AI replace developers? I gave it a try:
59. Let's see how far we can take this…
60. Let's see how far we can take this…
61. At this point, ChatGPT is lying through the teeth
62. Yay!
63. Now the tricky part…
64. But ChatGPT is easily convinced…
65. Chugging along…
66. And we're done! Little apple, little egg
67. Ouch…
69. AI PROOF!
70. Will AI replace developers? Visual Basic didn't make developers obsolete Outsourcing didn't make developers obsolete No-code systems didn't make developers obsolete AI won't make developers obsolete
71. Will AI replace developers? Development is so much more than producing code. AI will make programming more productive, but we will still require software engineering.
72. Vertrouwelijk Aangepast voor naam van het bedrijf Versie 1.0 Thank you! Vereniging Informatici Defensie - October 5, 2023 Midjourney - "a symposium for ministry of defense staff, with drones in attendance, sci-fi movie style" Ivo Jansch ivo@egeniq.com @ijansch