apidays Paris 2024 - AI In My API Gateway ... But Why?, Mathieu Ancelin, APIM

AI In My API Gateway ... But Why?
Mathieu Ancelin, CTO - Cloud at APIM

apidays Paris 2024 - The Future API Stack for Mass Innovation
December 3 - 5, 2024

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. APIDays 2024 AI in my API Gateway … But why ???
2. APIDays 2024 Mathieu ANCELIN CTO & co-founder @ Cloud APIM Developer @ SERLI Creator & lead @ Otoroshi @TrevorReznik mathieu-ancelin
3. APIDays 2024 Cloud APIM API Management & reverse proxy http as a service • managed otoroshi instances • serverless / gitops • authify / auth. enforcement • Webshield / WAF • LLM Endpoints https://www.cloud-apim.com @cloudapim cloud-apim
4. APIDays 2024
5. APIDays 2024 API Gateways • Expose your internal/external APIs to their consumers ◦ API Management ? • Apply the same level of control across all your APIs ◦ regardless of the underlying technology • Monitoring and observability • etc Can be hard to configure !
6. APIDays 2024 why this talk ? • I’m an API Gateway developer • I’m not an GenAI fanboy • I see more and more projects using AI libraries that tends to replicate API gateway behaviors • I’m convinced that consuming LLM APIs through an API Gateway is better for you ◦ ensure controlled use of AI/LLMs in your organization ◦ organization rules compliance ◦ legal compliance ◦ costs efficiency ◦ eventually can help you “configure” your gateway in fun ways ;) • Just focus on your product ◦ let us do the heavy lifting
7. APIDays 2024 https://github.com/MAIF/otoroshi
8. APIDays 2024 Otoroshi LLM Extension https://github.com/cloud-apim/otoroshi-llm-extension Connect, setup, secure and seamlessly manage LLMs using an Universal/OpenAI compatible API
9. APIDays 2024 Let’s start !
10. APIDays 2024 egress proxy pattern/forward proxy pattern internal apps internal apps internal apps
11. APIDays 2024 Unified interface • Multiple providers, same API ◦ can be vendor specific ◦ can be OpenAI compatible • Allows Your Applications to Work with Any Provider/Model Without Rewriting Everything ◦ lots of existing products already works with the OpenAI API ◦ you just have to learn one API and focus on your product
12. APIDays 2024 Provider supports • OpenAI • Azure OpenAI • Mistral • Anthropic • Ollama • Cloudflare • Scaleway • Cohere • Gemini • Groq • HuggingFace • OVH AI Endpoints • X.ai • etc lot of providers are supported without you worrying about it
13. APIDays 2024 API Resilience • Automatic Retries ◦ timeout management ◦ network errors ◦ API errors ◦ circuit breaker • Fallback providers ◦ local (ollama) ◦ remote • Model Load Balancing ◦ multiple instances of the same “local” model ◦ different remote models ◦ hybrid • Parallel calls
14. APIDays 2024 Observability and reporting Every request to an LLM provider generates data for further analysis • the provider used • the model used • the prompt • the response received • token consumption metrics • consumer identity ◦ apikey ◦ token ◦ connected user Who ? When ? What ?
15. APIDays 2024 Observability and reporting
16. APIDays 2024 Costs optimization • Manage token consumption with quotas ◦ highly flexible solution ◦ quotas, grouping and time window based on whatever you want in the http request • Cache ◦ simple ◦ semantic ▪ based on embeddings and vector search database • Reporting based on observability data
17. APIDays 2024 Costs optimization - tokens rate limiting
18. APIDays 2024 Costs optimization - semantic cache
19. APIDays 2024 Costs optimization - semantic cache
20. APIDays 2024 Prompts guardrails Optimize the Privacy of Your Organization's Data and the Security of Your Users Through a Set of "Guardrails" Applied to Prompts and/or Responses. Better legal and organization rules compliance • regex, characters count, words count, sentences count, contains, semantic contains • webhook • Generic LLM calls • pre-configured LLM ◦ no gibberish ◦ no secret leakage ◦ no PIF ◦ language moderation ◦ semantic match ◦ no hallucinations ◦ no gender bias ◦ no racial bias ◦ no personal health informations ◦ no toxic language ◦ no prompt jailbreak ◦ etc
21. APIDays 2024 Enhanced security Leverage the Full Range of Authentication and Authorization Mechanisms Provided by the Gateway • apikeys, jwt tokens, biscuit tokens • OAuth2, OIDC, LDAP, SAMLv2 • OPA Rego, RBAC, ACLs, contextual authorizations Handling Secrets for LLM Provider Access • the gateway can do it on the fly • can use secret vaults support for that
22. APIDays 2024 Enhanced security - secrets vaults
23. APIDays 2024 Enhanced security - secrets vaults }
24. APIDays 2024 Prompts engineering • Prompt contexts ◦ Global or Local Model Configuration ◦ Adding Contextual Information ◦ Adapting the Model to Your Organization ◦ Standardizing Responses • Prompt templating ◦ Simplifying the Creation of LLM-Based APIs ◦ No-Code Approach
25. APIDays 2024 Run tool_calls at the edge • Define your tool_calls function directly on the gateway ◦ using whatever language you want ◦ compiled to WASM • Run those functions directly on the gateway ◦ Avoid Complex Protocol Handling ◦ avoid code duplication across organization ◦ organization wide common functions usage ◦ optimizations across organization ◦ just focus on your product
26. APIDays 2024 Run tool_calls at the edge
27. APIDays 2024 Run tool_calls at the edge
28. APIDays 2024 Run tool_calls at the edge
29. APIDays 2024
30. APIDays 2024 And much more …
31. APIDays 2024 Conclusion
32. APIDays 2024 You can try the Otoroshi LLM Extension on https://www.cloud-apim.com
33. APIDays 2024 or on Clever Cloud https://www.clever-cloud.com/
34. APIDays 2024 Questions?