Tweet
Tweet

Slide 1

Slide 1 text

Thomas Vitale YOW! Conference December 2024 Concerto for Java and AI Building Production-Ready LLM Applications @thomasvitale.com

Slide 2

Slide 2 text

Systematic Thomas Vitale @thomasvitale.com

Slide 3

Slide 3 text

LLM RAG Prompting Embeddings Vector Stores Hallucinations Agents Generative AI Ph. Francisco Emilio Diaz @thomasvitale.com

Slide 4

Slide 4 text

One Buzzword To Rule Them All @thomasvitale.com

Slide 5

Slide 5 text

The WHY Factor @thomasvitale.com

Slide 6

Slide 6 text

The WHY Factor What problem does it solve? How ready is it for production? Yeah, but how about the DevEx? @thomasvitale.com

Slide 7

Slide 7 text

Generative AI @thomasvitale.com

Slide 8

Slide 8 text

Machine Learning Subset of Arti fi cial Intelligence Platform/Infrastructure Platform Engineers HTTP API Application Developer Model Training Model Inference ML Engineers Data Preparation Data Scientists @thomasvitale.com

Slide 9

Slide 9 text

If you like it, you should put an API on it @thomasvitale.com

Slide 10

Slide 10 text

Model Inference via HTTP APIs Application Model Inference Service Do you wanna build a snowman? HTTP Application Database Service DELETE * FROM HYPE; JDBC @thomasvitale.com

Slide 11

Slide 11 text

Java for AI-Infused Applications Integrations with Model Inference Platform/Infrastructure Platform Engineers Model Training Model Inference ML Engineers Data Preparation Data Scientists Application Developers Application @thomasvitale.com

Slide 12

Slide 12 text

Models and Inference Services @thomasvitale.com

Slide 13

Slide 13 text

Inference Services and LLMs How to choose? Managed Service Unmanaged Service Cloud On-Premises Proprietary Open Source @thomasvitale.com

Slide 14

Slide 14 text

opensource.org/ai @thomasvitale.com

Slide 15

Slide 15 text

ollama.com @thomasvitale.com

Slide 16

Slide 16 text

ollama.com @thomasvitale.com

Slide 17

Slide 17 text

ollama.com @thomasvitale.com

Slide 18

Slide 18 text

Privacy @thomasvitale.com

Slide 19

Slide 19 text

Chatbot @thomasvitale.com

Slide 20

Slide 20 text

Chatbot @thomasvitale.com “Legacy software companies adding an AI chatbot to their product" @andykreed

Slide 21

Slide 21 text

Text Classification @thomasvitale.com

Slide 22

Slide 22 text

Text Classification Application Model Service Classify HARMONY @thomasvitale.com

Slide 23

Slide 23 text

Resilience LLM Applications Resilience4J @thomasvitale.com

Slide 24

Slide 24 text

LLM Observability Generative AI Request Rate Errors Duration Prompt Evaluation Token Usage Context Window @thomasvitale.com

Slide 25

Slide 25 text

LLM Security Risks (1) OWASP Top 10 for LLM Prompt Injection Model Denial of Service Sensitive Information Disclosure OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/ @thomasvitale.com

Slide 26

Slide 26 text

Semantic Search @thomasvitale.com

Slide 27

Slide 27 text

Semantic Search From Keywords to Meaning Application Melancholic Embedding Model Melancholic [42…] LIKE ‘%melancholic%' SQL Store Vector Store [42…] @thomasvitale.com

Slide 28

Slide 28 text

Data Ingestion LLM Applications JobRunr Document Reader Document Transformer Document Writer @thomasvitale.com

Slide 29

Slide 29 text

Question Answering with Docs @thomasvitale.com

Slide 30

Slide 30 text

Question Answering with Docs Retrieval Augmented Generation Application Melancholic instrument? Embedding Model Melancholic instrument? [42…] Get Similar Documents Vector Store Model Question + Similar Documents @thomasvitale.com

Slide 31

Slide 31 text

Retrieval Augmented Generation @thomasvitale.com From Naive to Advanced

Slide 32

Slide 32 text

One Does Not Simply Test LLM Applications @thomasvitale.com

Slide 33

Slide 33 text

Structured Data Extraction @thomasvitale.com

Slide 34

Slide 34 text

Structured Data Extraction From Text to JSON Application Text Text to Structured JSON Model Database Save Structured JSON @thomasvitale.com

Slide 35

Slide 35 text

Speech Transcription From Speech to Text Application Audio Audio to Text Audio Model Chat Model Text to Structured JSON @thomasvitale.com

Slide 36

Slide 36 text

Image Processing From Image to Text Application Image Image to Text Image Model Chat Model Text to Structured JSON @thomasvitale.com

Slide 37

Slide 37 text

Hallucinations @thomasvitale.com

Slide 38

Slide 38 text

Data Validation JSON Schema Humans in the Loop Optional Values Mitigating hallucination risks @thomasvitale.com

Slide 39

Slide 39 text

@thomasvitale.com

Slide 40

Slide 40 text

LLM Security Risks (2) OWASP Top 10 for LLM Insecure Output Handling Excessive Agency Insecure Plugin Design OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/ @thomasvitale.com

Slide 41

Slide 41 text

Agents @thomasvitale.com

Slide 42

Slide 42 text

Agents Tools/Function Calling Application Is this instrument available? API Function Call Result Question Model Function Call Result @thomasvitale.com Function Call Request

Slide 43

Slide 43 text

1979 IBM

Slide 44

Slide 44 text

Going to Production @thomasvitale.com

Slide 45

Slide 45 text

“Friends don’t let friends write Dockerfiles!” - Josh Long @thomasvitale.com

Slide 46

Slide 46 text

Image pack build Cloud Native Buildpacks From source code to container image Cloud Native Buildpacks https://buildpacks.io @thomasvitale.com

Slide 47

Slide 47 text

Image pack build gradle bootBuildImage Cloud Native Buildpacks From source code to container image Cloud Native Buildpacks https://buildpacks.io @thomasvitale.com

Slide 48

Slide 48 text

Build & Deploy Cloud Native Buildpacks Kubernetes Service Binding Native Executables with GraalVM Going to Production @thomasvitale.com

Slide 49

Slide 49 text

Service Bindings for Spring AI @thomasvitale.com

Slide 50

Slide 50 text

@vitalethomas Ph. Francisco Emilio Diaz Composer Assistant https://github.com/ThomasVitale/concerto-for-java-and-ai

Slide 51

Slide 51 text

Thomas Vitale @thomasvitale.com thomasvitale.com Concerto for Java and AI Building Production-Ready LLM Applications https://github.com/ThomasVitale/llm-apps-java-spring-ai https://github.com/ThomasVitale/concerto-for-java-and-ai