Upgrade to Pro — share decks privately, control downloads, hide ads and more …

apidays Munich 2025 - Streamline & Secure LLM T...

Avatar for apidays apidays
July 09, 2025
0

apidays Munich 2025 - Streamline & Secure LLM Traffic with APISIX AI Gateway (API7)

Streamline & Secure LLM Traffic with APISIX AI Gateway
Yilia Lin, Technical Writer at API7

apidays Munich 2025 - Accelerate AI Use Cases with APIs
July 2 & 3, 2025

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Avatar for apidays

apidays

July 09, 2025
Tweet

More Decks by apidays

Transcript

  1. Agenda 01 02 03 Apache APISIX Overview APISIX AI Gateway

    Overview Proxy Multi-LLMs and Token -based Rate Limiting 04 Q&A
  2. About Speaker  Apache APISIX Committer  Technical Writer at

    API7.ai  LinkedIn: linkedin.com/in/yilialin/  GitHub: github.com/Yilialinn Yilia Lin
  3. Apache APISIX Overview • Donated to Apache Software Foundation by

    API7.ai in 2019 • Ultra High-Performance: > 23,000 single-core QPS • Low Latency: < 0.6 ms average delay • Lightweight Architecture: Decoupled control plane and data plane • High Scalability: >100 open-source plugins • Open-Source without Vendor Lock-in: Apache License 2.0
  4. The Rise of AI and New Challenges AI Application Characteristics

    • High-concurrency LLM Services • Token-based Pricing Model • Dynamic Scalability • Content Sensitivity New Challenges • Traffic Governance • Cost Optimization • Multi-Version Management • Content Security
  5. Configure Multi-LLMs and Implement Token-Based Rate Limiting  demo: https://app.storylane.io/share/cjpfweudrq1n

     doc: https://docs.api7.ai/hub/ai-proxy-multi#configure-instance- priority-and-rate-limiting