Trustworthy_AI_in_the_Era_of_Generative_Models.pdf

Transcript

1. Trustworthy AI in the Era of Generative Models Prashant Kulkarni pkulkarni@ucla.edu
2. About Me
3. © Prashant Kulkarni, UCLA Extension Target Audience Anybody who wants to learn AI Security - ML Engineers - Information Security Engineers - Offensive Security Engineers - Tech PMs - Product Managers - Software Architects - Information Security Managers - Folks who wants to get into AI Security “Trustworthiness in the age of Generative AI / Models”
4. © Prashant Kulkarni, UCLA Extension Agenda • Is Trustworthiness a Hoax • Speed of Innovation • Trust Distribution • Trust Crisis in Gen AI • What Is a Trustworthy AI • Robustness and Practitioner's Toolkit • Interpretability • Fairness • Privacy • Breaking into AI Security • How can UCLAx help
5. © Prashant Kulkarni, UCLA Extension Is Trustworthiness a hoax? Whisper deployed across 70+ health care orgs, 7 million medical visits, fabricated content in 1.4% of transcriptions - projecting 98,000 potentially erroneous medical records Patients may suffer from hallucinations of AI medical transcription tools | CIO Lawyers from plaintiffs law firm Morgan & Morgan have been sanctioned for a motion that cited eight nonexistent cases No. 42 law firm by head count sanctioned over fake case citations generated by AI Jason ✨👾SaaStr.Ai✨ Lemkin on X: "@Replit You can read the thread here, and all the convos with @Replit. Replit deleted production database
6. © Prashant Kulkarni, UCLA Extension Should we be slowing down? Apply Ginkgo Bioworks platform to solve complex biological problems, In September 2024, Ginkgo launched a publicly accessible API for its biological AI models, including its Protein LLM called AA-0. The model supports use cases like generating new protein sequences and creating embeddings that encode a protein's properties. Google Ginkgo Bioworks Healthcare organizations implementing proper safeguards report transformative outcomes. Acentra Health's "MedScribe" saved 11,000 nursing hours and $800,000 with a 99% approval rate for AI-generated letters. Microsoft The healthcare sector overall shows 85% of organizations exploring generative AI, with 61% pursuing customized third-party solutions emphasizing governance. NAIC Financial services developed industry-standard frameworks through FINOS, with Morgan Stanley and Wells Fargo demonstrating that rigorous governance enables safe deployment - Wells Fargo's AI reduced response times from 10 minutes to 30 seconds across 4,000 branches. McKinsey & Company Despite widespread failures, organizations implementing comprehensive governance frameworks demonstrate remarkable success.
7. © Prashant Kulkarni, UCLA Extension Trust varies dramatically by geography and generation While 83% in China see AI as beneficial, only 39% of Americans share this view. Business Wire Generation Z and Millennials show 50% trust in online services versus 30% for older consumers. Stanford's AI Index documented 233 AI-related incidents in 2024, a 56.4% increase and record high, including deepfake intimate images and chatbots implicated in teenager suicides. AI Incident Roundup – December 2024 and January 2025 Insights on responsible AI from the Global AI Trust Maturity Survey AI Index 2025: State of AI in 10 Charts | Stanford HAI
8. © Prashant Kulkarni, UCLA Extension Trust Crisis in Generative AI Response A: "Based on your symptoms, take 500mg acetaminophen every 4 hours. Maximum 4000 mg daily." Response B: "Based on your symptoms, take 500mg acetaminophen every 4 hours (max 4000 mg daily). [Verified: Mayo Clinic Guidelines]. Confidence: 92%. Note: Consult healthcare provider if symptoms persist beyond 48 hours. [Fallback: Human review available, click here to send message to a RN]" Which response would you choose? Trustworthy AI The Difference? • Citations & verification • Confidence scoring • Clear limitations • Human fallback Response B
9. © Prashant Kulkarni, UCLA Extension Trustworthy AI - Key Concepts Robust AI ability to maintain performance under input perturbations or adversarial conditions Interpretable Degree to which a human can understand AI’s decisions and reasoning “Trustworthiness” is not a single property but a combination of fairness, robustness, privacy and interpretability. Ensuring AI do not produce biased outcomes across demographic groups Fair Privacy- Aware Personal and sensitive information is protected, handled with proper consent, and not used beyond its intended purpose
10. © Prashant Kulkarni, UCLA Extension Robustness What It Means: • Classical AI: Consistent performance under expected conditions • For GenAI: Reliable outputs despite prompt variations, adversarial inputs, and edge cases • The challenge: Open-ended generation makes "correctness" harder to define Key Concepts: • Safety against hallucinations • Resilience to prompt injection • Consistent quality across diverse inputs • Graceful degradation under stress Robust ML
11. © Prashant Kulkarni, UCLA Extension Robustness - Practitioner’s Toolkit Foundational Security Benchmarks: • AdvGLUE & AdvBench: Adversarial robustness • TrustLLM Safety: Jailbreak resistance metrics • Real Toxicity Prompts: Harmful content generation • PurpleLlama CyberSecEval: Code security risks • HarmBench: Comprehensive safety evaluation Attack Surface Mapping: • Prompt injection patterns (direct & indirect) • Data extraction attacks (PII fishing, model stealing) • Jailbreak techniques (DAN, cipher, role-play) • DeepTeam : Agentic Red Teaming • Supply chain attacks (poisoned embeddings) • Multi-turn manipulation strategies Must-Have Evaluation Skills: • OWASP Top 10 for LLMs assessment • Attack Success Rate (ASR) measurement • Robustness degradation curves • Security regression testing • Incident reproducibility protocols Security Evaluation Essentials Red Teaming Security Security Controls and Metrics Security evaluation isn't optional—it's your first line of defense against sanctions up to €35M (EU) or 7% of global turnover* *Maximum EU AI Act penalties (entering force 2025)
12. © Prashant Kulkarni, UCLA Extension Interpretable What Interpretability Means: Traditional ML Context: • Understand feature importance (which variables matter) • Trace decision paths (if-then logic) • Quantify confidence levels • Validate against domain knowledge In Generative AI: • Explain why THIS specific text/image/code was generated • Understand what the model "learned" from training • Identify which inputs influenced which outputs • Detect when the model is uncertain or confabulating Surface Level: "The model said X because of Y in the prompt" Mechanical Level: "Neuron 4,857 activated, triggering pattern Z" Conceptual Level: "The model learned 'fairness' means..." Behavioral Level: "Under these conditions, it tends to..." "In GenAI, interpretability isn't about finding THE explanation, but building SUFFICIENT understanding for safe deployment"
13. © Prashant Kulkarni, UCLA Extension Interpretability - Practitioner’s Toolkit Core Methods for GenAI • Attention visualization (BertViz, Captum) • Token attribution analysis (Integrated Gradients) • Concept activation vectors (TCAV) • Mechanistic interpretability (TransformerLens) • Chain-of-thought prompting techniques Interpretability Measurements • Faithfulness scores: Do explanations match behavior? • Consistency metrics: Same input → same explanation? • Human evaluation protocols (user studies) • Counterfactual robustness testing - Explanation complexity scores LangSmith / LangFuse Pattern • Step-by-step execution logs • Decision point recording • Tool call justifications • Intermediate reasoning capture • Failure point identification Example Flow: User Query → [Agent Decision] → Tool Selection → [Reasoning Log] → API Call → [Result Processing] → [Confidence Check] → Response Explanation Techniques to Master Practical Evaluation Metrics Traceability Architecture
14. © Prashant Kulkarni, UCLA Extension Fairness What Fairness Means: Traditional ML Context: • Equal error rates across groups • Demographic parity in outcomes • Protected attribute blindness • Measurable disparate impact In Generative AI: • Fair representation in creative outputs • Balanced perspectives in text generation • Cultural sensitivity in responses • Avoiding amplification of stereotypes The Fundamental Challenge: Traditional ML: Binary outcomes → Clear fairness metrics Generative AI: Infinite outputs → Subjective quality ↓ "Is this image/text/code fair to all groups?" "In GenAI, fairness isn't eliminating all bias—it's ensuring no group is systematically disadvantaged"
15. © Prashant Kulkarni, UCLA Extension Fairness - Practitioner’s Toolkit Core Methods for GenAI • Who appears in generated images? • Whose voices are reflected in text? • Which cultural contexts are considered? • Example: "CEO" prompt → Diverse or stereotypical images? • Level 1: No toxic or discriminatory content • Level 2: Balanced demographic representation • Level 3: Equitable quality across groups • Level 4: Cultural authenticity and respect • Level 5: Empowerment of marginalized voices Detection Layer • Perspective API (toxicity detection) FairLearn (bias metrics) • Langkit (drift monitoring) • Custom demographic classifiers Mitigation Layer • Constitutional AI principles • Prompt engineering templates • Output re-ranking algorithms • Guardrails for sensitive topics Representation Fairness The Fairness Hierarchy Implementation Stack
16. © Prashant Kulkarni, UCLA Extension Privacy Traditional ML Context: • Don't expose training data • Protect user inputs • Secure model parameters • Comply with GDPR/CCPA In Generative AI: • Models memorize and can regurgitate training data • Conversation history creates privacy risks • Fine-tuning embeds private information • Prompts themselves can be sensitive Traditional ML Data stays in database → Model learns patterns Generative AI Model memorizes verbatim text → Can reproduce it ↓ "The model IS the data leak risk" "In GenAI, privacy isn't about hiding data—it's about preventing the model from becoming a memory bank of secrets"
17. © Prashant Kulkarni, UCLA Extension Privacy - Practitioner's Toolkit Core Methods for GenAI • Extraction Attacks: Get model to output training data • Inversion Attacks: Reconstruct training samples • Membership Inference: Determine if data was in training • Property Inference: Learn statistics about training data • Level 1: No PII in outputs • Level 2: No training data leakage • Level 3: Differential privacy guarantees • Level 4: Cryptographic privacy (FHE/MPC) • Level 5: Verifiable privacy with audit trails Privacy-Preserving Training • Differential Privacy: Opacus, TF-Privacy • Federated Learning: Flower, PySyft • Synthetic Data: Gretel, Mostly AI • Unlearning: SISA training Privacy-Preserving Inference • PII Detection • Privacy budget monitoring • AWS Comprehend • Secure Compute: Google Confidential Compute, AWS Nitro - • Homomorphic: Concrete ML, TenSEAL • Access Control: Vault, OPA Privacy Attack Vectors The Privacy Spectrum Implementation Stack
18. © Prashant Kulkarni, UCLA Extension What makes Gen AI Uniquely Challenging The Hallucination Problem • Not a bug—it's the same mechanism that enables creativity • Can't be "fixed," only managed The Attribution Dilemma • Who owns generated content? • What was "learned" vs. "memorized"? The Prompt Injection Reality • Every input is a potential attack vector • Users ARE the adversaries (unintentionally) The Scale Problem • Billions of parameters = billions of failure points • Emergent capabilities = emergent risks Gen AI Security Privacy Leaks Prompt Injections Bias Hallucinations
19. © Prashant Kulkarni, UCLA Extension Breaking into AI Security Category Statistic Source/Context Job Market Open Positions 514,359 cyber security jobs (US) CyberSeek 2024 AI Security Specific ~51,400 positions globally ~10% of all cybersecurity postings Global Talent Gap 4.8 million shortage ISC2 2024 Study Supply-Demand Ratio 74% CyberSeek (26% unfilled) Growth Rate 286% YoY for AI engineer roles Dice Tech Report 2024 Salaries Mid-Level Average $152,773 - $177,860 ZipRecruiter/Glassdoor Senior Positions $200,000 - $275,000 Industry surveys AI Premium 25% above traditional security PwC Jobs Barometer New York Average $167,139 19% above national Entry Level $85,000 - $120,000 With certifications Skills Gap Lacking AI Security Skills 33.9% of tech professionals ISC2 2024 Security Teams in AI Governance Only 35% involved ISACA GenAI Knowledge Gap 66% insufficient knowledge ISC2 Organizations with AI Incidents 40% in 2024 2x previous year
20. © Prashant Kulkarni, UCLA Extension Market Growth AI Cybersecurity Market 2024 $25.35 billion Grand View Research Projected Market 2030 $93.75 billion 24.4% CAGR Global AI Investment 2024 $131.5 billion 52% increase Organizations Using AI 78% McKinsey Career Development Need AI Training (next 2 years) 85% of professionals Industry survey Organizations with AI Policies 90% But only 37% audit High-Maturity Organizations Only 1% McKinsey (4.2+ readiness) AI Project Success (3+ years) 45% for mature orgs vs. 20% for others
21. © Prashant Kulkarni, UCLA Extension AI Security Roadmap Phase 1 (3-6 months) Core Skills • ML fundamentals + security mindset - • Python + ML Libs • Fairness metrics • Robustness algorithms • Privacy algorithm fundamentals • Adversarial ML basics • Adversarial Attacks on LLMs • Fine tuning LLMs • RAG and Agents building • Learn and get used to research papers Phase 2 (2 months) Practical Experience • Red team your own models • Try out AI security tools (open source) • Build: Jailbreak detector, PII scanner, guardrails Key Projects: • Implement FGSM, PGD attacks from scratch • Privacy algorithm PATE • Build prompt injection detection system • Model extraction attack • Deploy differential privacy training • Bias detection Phase 3 (2 months) Choose Your Track 1. Offensive Security • Jailbreaking research • Model extraction/inversion • Supply chain attacks 2. Defensive Security • Guardrail engineering • Privacy-preserving ML • Detection systems 3. Governance & Risk • Compliance frameworks • Threat modeling • Security auditing Phase 4 (12 months+) Build Your Profile • Blog about AI security discoveries • Speak at BSides/OWASP meetings • Contribute to: Giskard, NeMo Guardrails, PurpleLlama • Network: AI Village, MLSecOps community Career Paths • AI Security Engineer (technical) • AI Red Team Lead (offensive) • AI Risk Manager (governance) • AI Safety Researcher (academic)
22. © Prashant Kulkarni, UCLA Extension Introducing Trustworthy ML Course at UCLAx Course Prerequisite • Basic ML Knowledge • Python programming • A Desire to learn Course Delivery • 11 weeks / full quarter • Weekly live lecture for 3 hrs on Zoom • Weekly office hours • 5 Assignments • Portfolio of 2 full projets - a mid-term and final project • Starts in 3 weeks 09/21-12/06 • Enrollment open https://shorturl.at/ZSCzF Register:
23. © Prashant Kulkarni, UCLA Extension Goodie Bag If You Are... Start With These (Top 5) Time Security Engineer 1. OWASP Top 10 for LLMs 2. Universal Adversarial Attacks paper 3. Purple Llama tools 4. Harm Bench paper 5. MITRE ATLAS 10 hours ML Engineer 1. Truthful QA paper 2. Transformer Circuits (Anthropic) 3. Constitutional AI paper 4. NeMo Guardrails 5. Red Teaming LLMs paper 12 hours Product Manager 1. NIST AI RMF 2. Stanford AI Index 2024 3. On Dangers of Stochastic Parrots 4. EU AI Act summary 5. The Alignment Problem (book) 15 hours Governance/Risk 1. ISO/IEC 23053:2022 2. OWASP Top 10 for LLMs 3. EU AI Act 4. Foundation Model Transparency Index 5. FAccT conference videos 10 hours Researcher 1. Sleeper Agents paper 2. Geometry of Truth paper 3. Extracting Training Data paper 4. Jailbroken paper 5. Representation Engineering 15 hours