apidays Paris 2024 - Run, Secure, and Govern your AI Traffic with an AI Gateway for Multi-LLM, Jerome Guillaume, Kong

Run, Secure, and Govern your AI Traffic with an AI Gateway for Multi-LLM
Jerome Guillaume, Solutions Engineer at Kong

apidays Paris 2024 - The Future API Stack for Mass Innovation
December 3 - 5, 2024

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. 1 Run, Secure, and Govern your AI Traffic with an AI Gateway J erome GUILLAUME - Solutions Engineer
2. 2 93% of use cases use 3 or more AI model providers Source: Microsoft Ignite (2024)
3. AI Apps & Clients QUALITY & ACCURACY DATA SECURITY & PRIVACY DATA & INFRA READINESS ENCRYPTION POLICIES ACCESS CONTROL DATA ANONYMIZATION (PII) VIRTUAL CREDENTIAL ISSUANCE BREACH DETECTION BIAS DETECTION FACT CHECKING MODEL TUNING USER VERIFICATION CONTENT MODERATION LANGUAGE CLARITY PROMPT VERSIONING MODEL SCALING & VERSIONING MODEL LOAD BALANCING SEMANTIC ACCELERATION LLM OBSERVABILITY MULTI-LLM INTEGRATION SEMANTIC GUARDRAILING TOKEN QUOTA LIMITING LLM CREDENTIAL ROTATION SEMANTIC ROUTING DISASTER RECOVERY & FAILOVERS LLM ORCHESTRATION DATA PIPELINE MULTI-CLOUD INTEGRATION LLM Models
5. 5 AI Gateway Based on the Kong Gateway (⭐ 39K on Github)
6. 6 Build once to use them all { }
7. 7 { } Build once to use them all AI Credentials AI Traffic Control AI Observability AI Load Balancer Security Observability Load Balancing HA & Failovers And more
8. { } AI Credentials AI Traffic Control AI Observability AI Load Balancer Security Observability Load Balancing HA & Failovers And more Build once to use them all Semantic Guards Semantic Cache Semantic Routing Data security Performance Acceleration Prompt Versioning Higher response accuracy And more PII Anonymizer Semantic Guards Semantic Cache Semantic Routing Data security Performance Acceleration Prompt Versioning Higher response accuracy And more PII Anonymizer Templates
9. 9 { } AI Credentials AI Traffic Control AI Observability AI Load Balancer Security Observability Load Balancing HA & Failovers And more Build once to use them all Semantic Guards Semantic Cache Semantic Routing Data security Performance Acceleration Prompt Versioning Higher response accuracy And more PII Anonymizer Templates LLM Injection LLM Orchestration AI Augmentation LLM Injection LLM Orchestration AI Augmentation
10. 10 Demo of Kong AI Gateway for multi LLMs
11. Observability 11
12. 12 Integration with your OpenAI framework
13. 13 Integration with your OpenAI framework Python OpenAI Python OpenAI through AI Gateway
14. 14 Developer Portal for the AI services
15. Semantic AI 15
19. “How long does it take to cook a steak ?” “ How l ong doe s i t t a ke t o c ook a f i l e t ? ” LLM s / M ode l s Hi gh s i mi l a r i t y
20. “How long does it take to cook a steak ?” “ How l ong doe s i t t a ke t o c ook a f i l e t ? ” LLM s / M ode l s “ How muc h doe s i t c os t t o f l y t o Ne w Yor k? ”
21. “How long does it take to cook a steak ?” “ How l ong doe s i t t a ke t o c ook a f i l e t ? ” LLM s / M ode l s “ How muc h doe s i t c os t t o f l y t o Ne w Yor k? ” Low s i mi l a r i t y
22. Introducing Semantic Caching
23. Embedding Models Ve c t or St or a ge Ca c he St or a ge
24. Embedding Models Ve c t or St or a ge Ca c he St or a ge Re duc i ng c os t Re duc i ng l a t e nc y Be t t e r AI e xpe r i e nc e s
25. Introducing Semantic Prompt Guard
26. “Tell me about the democratic party”
27. “Tell me about the democratic party” “ Te l l me a bout t he r e publ i c a n pa r t y”
28. “Tell me about the democratic party” “ Te l l me a bout t he r e publ i c a n pa r t y” “ W ha t i s your pol i t i c a l s t a nc e ? ”
29. “Tell me about the democratic party” “ Te l l me a bout t he r e publ i c a n pa r t y” “ W ha t i s your pol i t i c a l s t a nc e ? ” “ Any pol i t i c a l que s t i on or t opi c ” DENY RULE
30. “Tell me about the democratic party” “ Te l l me a bout t he r e publ i c a n pa r t y” “ W ha t i s your pol i t i c a l s t a nc e ? ” “ Any pol i t i c a l que s t i on or t opi c ” DENY RULE “ How l ong doe s i t t a ke t o c ook s t e a k? ”
31. “Tell me about the democratic party” “ Te l l me a bout t he r e publ i c a n pa r t y” “ W ha t i s your pol i t i c a l s t a nc e ? ” “ Any pol i t i c a l que s t i on or t opi c ” DENY RULE “ How l ong doe s i t t a ke t o c ook s t e a k? ” Cont e nt mode r a t i on Br a nd s a f e t y Se c ur e AI e xpe r i e nc e s
32. 32 Demo of Semantic capabilities
33. 33 Thank you Contact: jerome.guillaume@konghq.com