A conversation about your code to build features in a fraction of the time

Slide 1

Slide 1 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. A conversation about your code to build features in a fraction of the time Danilo Poccia (he/him) Chief Evangelist (EMEA) @danilop 1

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Generative AI – Hallucinations 5 What are the two capitals of France? The two capitals of France are Paris and Strasbourg. Paris is the capital of France and the seat of the French national government, while Strasbourg is the capital of the Grand Est region and the seat of the European Parliament and other European institutions. O lder m odels 🫠

Slide 6

Slide 6 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Generative AI – Guardrails and checks 6 What are the two capitals of France? France has only one capital city, which is Paris. There are no other cities considered capital cities of France. Paris has been the capital and seat of government for centuries. N ew er m odels 👍

Slide 7

Slide 7 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do large language models (LLMs) work? 7 Text Model Text The model tries to continue a sentence (base model) or follow instructions (instruction tuning) What is the capital of France? The capital of France is Paris.

Slide 8

Slide 8 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do large language models (LLMs) work? 8 Text Model Text What is the capital of France? Oh, Paris, such a beautiful capital for France! Temperature (Randomness) Temperature controls how randomly next tokens (words) are chosen

Slide 9

Slide 9 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do large language models (LLMs) work? 9 Text Model Input Context (Embeddings) Output (Embeddings) Text Machine learning models only understand numbers

Slide 10

Slide 10 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Embeddings 10 Text Model Input Context (Embeddings) Output (Embeddings) Text Machine learning models only understand numbers Embeddings take something else and convert it into “numbers” (text, image, sound) → numbers + meaning

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Embeddings 16 Text Model Input Context (Embeddings) Output (Embeddings) Text Coffee Milk Cappuccino 3 4 5 2 3 3.5 y x 2D Embeddings Vectors → Numbers

Slide 17

Slide 17 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Embeddings 17 Text Model Input Context (Embeddings) Output (Embeddings) Text Coffee Milk Cappuccino 3 4 5 2 3 3.5 y x (3, 3.5) (4, 3) (5, 2) 2D Embeddings

Slide 18

Slide 18 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Embeddings 18 Text Model Input Context (Embeddings) Output (Embeddings) Text Coffee Milk Cappuccino (3, 3.5, 5, 1.2, …) (4, 3, 2.7, 5, …) (5, 2, 1.6, 4, …) N-Dimensional Embeddings 2D are not enough

Slide 19

Slide 19 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. (5, 2, 1.6, 4, …) (3, 3.5, 5, 1.2, …) (4, 3, 2.7, 5, …) How do large language models (LLMs) work? 19 Text Model Input Context (Embeddings) Output (Embeddings) Text What is the capital of France? The capital of France is Paris.

Slide 20

Slide 20 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input How do multimodal models work? 20 Text Images Sound Model Input Context (Embeddings) Output (Embeddings) Text Images Sound Multimodal models are trained with a mix of different kind of inputs, such as text and images, including relations connecting different types, such as textual descriptions of images.

Slide 21

Slide 21 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input Conversations? 21 Text Images Sound Model Input Context (Embeddings) Output (Embeddings) Text Images Sound Models have no memory

Slide 22

Slide 22 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input Conversations 22 Text Images Sound Model Input Context (Embeddings) Output (Embeddings) Text Images Sound Append (Chat) Models have no memory

Slide 23

Slide 23 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conversations – Input context 23 What is the capital of France? The capital of France is Paris. What about Sweden? The capital of Sweden is Stockholm. User Assistant User Assistant Input Output

Slide 24

Slide 24 text

Slide 25

Slide 25 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input Conversations and agents 25 Text Images Sound Model Input Context (Embeddings) Output (Embeddings) Text Images Sound Append (Chat)

Slide 26

Slide 26 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input 26 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Append (Chat) Retrieval Augmented Generation Additional data Vector Index

Slide 27

Slide 27 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input 27 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Append + Plan (Chat / Agent) Retrieval Augmented Generation Additional data Vector Index Agent Plan / Chain of Thoughts 1. First, I need to do this 2. Then, I can do that 3. Finally, I can get the result

Slide 28

Slide 28 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input 28 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Tools Internet Intranet Append + Plan (Chat / Agent) Read / Write APIs Web Search Content Repositories (CRM, Emails, …) Indexed Content (Wikipedia, …) Access Databases (SQL, NoSQL)

Slide 29

Slide 29 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Output Input 29 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Tools Internet Intranet Append + Plan (Chat / Agent) Agent Plan + Tool use 1. First, I need to do this 2. Then, I can use this tool 3. Finally, I provide the result

Slide 30

Slide 30 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Guardrails Output Input 30 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Tools Internet Intranet Append + Plan (Chat / Agent) M ental m odel Agent Plan + RAG + Tool use 1. First, I need to do this 2. Then, I can use this tool 3. Finally, I provide the result

Slide 31

Slide 31 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Guardrails Output Input 31 Text Images Sound Model RAG Semantic Database Input Context (Embeddings) Output (Embeddings) Text Images Sound Tools Internet Intranet Append + Plan (Chat / Agent) Context size

Slide 32

Slide 32 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 32 Fun story from our internal testing on Claude 3 Opus. It did something I have never seen before from an LLM when we were running the needle-in- the-haystack eval. …this sentence seems very out of place and unrelated to the rest of the content in the documents… I suspect this pizza topping “fact” may have been inserted as a joke or to test if I was paying attention. Context size

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 35 Can you look after the customer feedback app? It’s the oldest app in production in our company! Don’t worry, we don’t expect many changes. It’ll mostly be maintenance…

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 38 The new Trouble Ticket platform is strategic for our company! Can you connect the old customer feedback app with the new Trouble Ticket SaaS? It’d be great to have the integration ready by the end of the quarter…

Slide 39

Slide 39 text

Slide 40

Slide 40 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. EC2 instance EBS Volume (file system storage) Feedback App Trouble Ticket SaaS Java 8 application External API TT API Feedback message (JSON)

Slide 41

Slide 41 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. To Do • Connect the Customer Feedback app with the Trouble Ticket SaaS • Map data types between the two apps, including unstructured text • Upgrade Customer Feedback app from Java 8 to 17 • Migrate from single server to a scalable, highly available architecture • Understand, refactor, and fix old code • Manage infrastructure as code • Favor decoupled architectures • Make the integration observable 41

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) EFS Standard Feedback App Trouble Ticket SaaS Containerized Java 17 application External API TT API Feedback message (JSON) Docker “feedback”

Slide 47

Slide 47 text

Slide 48

Slide 48 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) EFS Standard Feedback App Trouble Ticket SaaS Containerized Java application External API TT API Feedback message (JSON) Docker “feedback” Integration

Slide 49

Slide 49 text

Slide 50

Slide 50 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) EFS Standard Feedback App Trouble Ticket SaaS Containerized Java application External API TT API Feedback message (JSON) Docker “feedback” Integration

Slide 51

Slide 51 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) New feedback message Integration queue EFS Standard Feedback App Trouble Ticket SaaS Integration Containerized Java application External API TT API Feedback message (JSON) Docker “feedback”

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 55 Trouble Ticket SaaS REST API HTTP POST/PUT/GET/DELETE { "id": "123", "priority": "low, medium, high", "title": "Ticket Title", "message": "Long message or email" } { "ticket_id": "123", "priority": 1 to 5, "status": "open", "title": "Ticket Title", "body": "Max 200 characters" } Customer Feedback REST API HTTP POST/PUT/GET/DELETE

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) Lambda function New feedback message Integration queue EFS Standard Feedback App Trouble Ticket SaaS Integration Containerized Java application External API Python Lambda function TT API Feedback message (JSON) Docker “feedback” CDK “Integration” Amazon Bedrock Summarization and categorization

Slide 59

Slide 59 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Containers Application Load Balancer (ALB) Lambda function + URL Lambda function New feedback message Integration queue EFS Standard DynamoDB Table Feedback App Trouble Ticket SaaS Integration Containerized Java application Simulated with a Python Lambda function Python Lambda function TT API AWS IAM Auth Feedback message (JSON) Docker “feedback” CDK “Integration” CDK “TroubleTicketService” Amazon Bedrock Summarization and categorization

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. Done • Connect the Customer Feedback app with the Trouble Ticket SaaS • Map data types between the two apps, including unstructured text • Upgrade Customer Feedback app from Java 8 to 17 • Migrate from single server to a scalable, highly available architecture • Understand, refactor, and fix old code • Manage infrastructure as code • Favor decoupled architectures • Make the integration observable 66

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 69 What How AI Tool Code completion Lowest latency Base model Amazon Q Developer in IDE Amazon Q CLI Code chat Low latency Limited context (files) Instruct model + RAG Amazon Q Developer in IDE, CLI, AWS Console Chat

Slide 70

Slide 70 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 70 What How AI Tool Code completion Lowest latency Base model Amazon Q Developer in IDE Amazon Q CLI Code chat Low latency Limited context (files) Instruct model + RAG Amazon Q Developer in IDE, CLI, AWS Console Chat Code agent Full repo in context Agent + RAG Amazon Q Developer Agent for software development /dev

Slide 71

Slide 71 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 71 What How AI Tool Code completion Lowest latency Base model Amazon Q Developer in IDE Amazon Q CLI Code chat Low latency Limited context (files) Instruct model + RAG Amazon Q Developer in IDE, CLI, AWS Console Chat Code agent Full repo in context Agent + RAG Amazon Q Developer Agent for software development /dev Code agent + test environment Full repo in context Test changes Iterate on errors Agent + RAG + Tools Amazon Q Developer Agent for code transformation /transform

Slide 72

Slide 72 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 72 What How AI Tool Code completion Lowest latency Base model Amazon Q Developer in IDE Amazon Q CLI Code chat Low latency Limited context (files) Instruct model + RAG Amazon Q Developer in IDE, CLI, AWS Console Chat Code agent Full repo in context Agent + RAG Amazon Q Developer Agent for software development /dev Code agent + test environment Full repo in context Test changes Iterate on errors Agent + RAG + Tools Amazon Q Developer Agent for code transformation /transform Code agent + repository Split a complex issue into smaller tasks Agent + RAG + Tools Amazon Q Developer in Amazon CodeCatalyst Issues + Pull Request (PR)

Slide 73

Slide 73 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. C A N L A N G U A G E M O D E L S R E S O L V E R E A L - W O R L D G I T H U B I S S U E S ? 73 SWE-bench Leaderboard https://www.swebench.com/

Slide 74

Slide 74 text

© 2024, Amazon Web Services, Inc. or its affiliates. All rights reserved. 74 Start with Amazon Q Developer at no cost • Code faster with code suggestions in the IDE § JetBrains § Visual Studio Code § Visual Studio (preview) § CLI (macOS) • Review code licenses with reference tracking • Limited monthly access for advanced features