From Prompt to Prediction: Understanding LLM Outputs

Transcript

1. FamilyID=Office_ArchiveTorn From Prompt to Prediction: Understanding LLM Outputs Sena Yakut, 2025
2. About me Sena Yakut, 2025 Cloud Security Architect All links about me
3. Understanding LLMs
4. LLMs bring humans, data, and compute power into the same room.
5. big data + big model + big computation. the world of words and meaning trained systems that learn to predict and generate Understanding LLMs
6. LLMs bring humans, data, and compute power into the same room. How do they actually work? Data Tokenization Transformers (Neural Network) Training Fine Tuning Inference Post processing
7. Data Collecting high-quality data is a huge part of building a cool LLM. The main goal → Use all of the clean internet. BUT It’s not that simple.
8. Data • Public web data: Websites, forums, Wikipedia, blogs, etc. • Books and articles: Digitized texts from open libraries or licensed publishers. • Code repositories: For coding models (e.g., GitHub, open- source datasets).
9. Data • Academic and scientific papers: To include structured, factual information. • Conversational data: Chat logs or dialogues (when available and anonymized).
10. Data Data Filtering Remove duplicates, spam, and irrelevant content. Filter out toxic, biased, or unsafe language. Normalize text (e.g., fixing encoding issues, removing special symbols). Exclude personal or sensitive information.
11. Tokenization Helps an LLM read and understand text -> small chunks called tokens “Cybersecurity” → ["Cyber", "security"] or sometimes → ["Cy", "ber", "security"] Are single letters stored as tokens — for typos or rare words? Yes, kind of. If the model doesn’t recognize a full word, the tokenizer breaks it into smaller known pieces, even letters. Example: “cybeersecurity” (typo) → ["cy", "bee", "r", "security"] So every letter can be a fallback token if needed.
12. Tokenization Are homonyms (same writing, different meaning) the same token? Yes. Because tokenization only looks at the letters, not meaning. Example: “bank” (money) and “bank” (river) → same token. Meaning comes later during training, when the model learns context.
13. Tokenization Are homophones or synonyms (different writing, same meaning) the same token? No. “see” and “sea” → different tokens. “car” and “automobile” → different tokens. Even if they mean the same thing, tokenization doesn’t know that. Meaning is learned on training / transformers.
14. Tokenization
15. LLMs bring humans, data, and compute power into the same room. Transformers • Transformers are the core architecture behind modern LLMs like GPT, LLaMA, Claude, and Gemini. • They were introduced in 2017 by Google researchers in the paper “Attention Is All You Need.” • Just attention mechanisms are enough to understand relationships between words.
16. Transformers Word Attention Focus (→ “it”) Explanation The low Not meaningful cat high “it” most likely refers to “cat” chased low Action, not reference the low neutral dog medium somewhat related because low connector word The cat chased the dog because it was scared. They’re useful. But not the full picture.
18. LLM Models are everywhere. 1. Define the goal: What do you need, chat, code, text analysis, or reasoning: Each task might need a different model. 2. Check data alignment: Does the model understand your domain (medical, legal, IoT, etc.)? 3. Balance performance & cost: Choose the smallest model that meets your needs efficiently. 4. Privacy & compliance: If your data is sensitive, prefer on-prem or private LLM options. 5. Evaluate accuracy: Test with your own prompts and metrics, hallucinations are real!
19. LLMs are everywhere.
20. LLMs are everywhere. • llm-stats.com • artificialanalysis.ai Think of it like picking a car, you don’t always need a race car, sometimes a compact one is faster, cheaper, and easier to park.
21. LLMs are everywhere. We should not directly believe LLM outputs → they don’t know, they predict. Understanding that difference is how we stay in control.
22. LLM Hallucination Examples ChatGPT and Avianca Airlines Case • A lawyer used ChatGPT to find legal cases for a lawsuit against Avianca Airlines. • ChatGPT invented fake court cases that sounded real. • The lawyer didn’t verify them and gave them to the judge.
23. LLM Hallucination Examples
24. LLM Hallucination Examples Microsoft Ottawa Food Bank Incident • A Microsoft travel article about Ottawa Food Bank mistakenly listed a food bank as a tourist attraction. • It even said, “Consider going into it on an empty stomach” • Microsoft removed the article and said it was an error caused by poor human oversight in the content process.
25. LLM Hallucination Examples Technical – Package Hallucinations (Slopsquatting) • An LLM suggests or uses a software package/library name that doesn’t actually exist. • An attacker pre-registers that made-up name on a public package repository (e.g., PyPI, npm) and fills it with malicious code. • A developer using the AI suggestion installs the package, unknowingly introducing malware or a supply-chain vulnerability.
26. LLM Hallucination Examples Technical – Package Hallucinations (Slopsquatting)
27. LLM Hallucination Examples
28. LLM Hallucination Examples
29. LLM Coding • Secure code best practices, • Hardcoded credentials, • Lots of log details, • Broken access issues, • Write prompts detailed, step by step. • Review the code after fix / new feature. • Think about infrastructure.
31. Prompt Influence When people say... “It doesn’t understand me.” “It keeps giving the wrong answer.” “It doesn’t say what I mean.” What’s really happening It’s not that the LLM “doesn’t understand.” It’s that the prompt doesn’t guide it clearly enough. LLMs don’t read minds, they follow patterns.
32. Prompt Influence Topic: Describe the ocean Specificity Describe the ocean as if you’re trying to inspire people to protect it. →Specific prompts give specific emotions. Tone Control Describe the ocean like a scientist. → Tone words (“like a poet,” “like a scientist,” “like a child”) change the personality.
33. Prompt Influence Topic: Describe the ocean Expertise Level Explain the ocean to a 5-year-old. → Audience decides vocabulary and depth. And More… • Add a perspective: “Describe the ocean from a sailor’s point of view.” • Add a limit: “Describe the ocean in under 10 words.” • Add a format: “Write it as a tweet.” • Add a contrast: “Describe the ocean and the desert in one paragraph.” • Add a creative twist: “Describe the ocean as if it were an AI learning emotions.”
34. Prompt Influence
35. Prompt Influence Reusable Prompt Template: Topic: {subject} Goal: {what you want to achieve} Tone: {friendly | poetic | analytical | humorous | emotional} Audience: {child | student | professional | public} Structure: {bullets | story | comparison | Q&A | quote} Add-ons: - {perspective or persona} - {length or format limit} - {emotion or creativity cue} Word limit: {optional}
37. LLM Confidence ≠ LLM Correctness LLM confidence looks like: • Assertive tone (“clearly”, “definitely”) • No sign of doubt • Still possibly wrong!
38. How to Read an AI Answer How to Read an AI Answer • Don’t trust tone -> check the facts • Ask: “How do you know?” • Ask for sources or reasoning • Rephrase your question and compare • If it sounds too confident, ask it to slow down and explain
39. How to Recognize AI-Generated Content Language tells clues: • Too perfect grammar or symmetry • Generic phrases (e.g. “In today’s fast-paced world…”) • Overuse of transitions (“Moreover”, “In conclusion”) • Repeated patterns, smooth but emotionless tone • No real “voice” or perspective
40. Thank You!