Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Concerto for Java and AI - Building Production-...

Concerto for Java and AI - Building Production-Ready LLM Applications (YOW! Australia 2024)

Imagine you're a music composer struggling to find inspiration for a pivotal movie scene. Then, you remember you’re also a software engineer, and the solution becomes suddenly obvious. Join me in this session, where I'll demonstrate how I enhanced my music composition process by harnessing the power of Java and AI.

This talk will discuss the core architectural patterns for introducing AI capabilities into an existing software system, exploring use cases like text classification, structured data extraction, semantic search, and agentic tools. The Java ecosystem is getting more and more capabilities for building AI applications. But are they ready for production? Are there any gaps?

Throughout the session, I’ll build a "composer assistant" application using Spring AI to showcase how to make an LLM application production-ready. Is the developer experience affected when working locally with language models? How is observability different when it comes to tokens? Can we ensure resilience across the many integrations orchestrated by the AI? What strategies are available for deploying LLM applications?

In a final twist, you’ll choose which movie scene to score, and I’ll compose and perform the music live for it, supported by AI. Will it meet the mark? There’s only one way to find out: join me in exploring the practical side of AI applications, where Java and Generative AI offer tangible solutions to real-world use cases. Aaaaand action!

Thomas Vitale

December 17, 2024
Tweet

More Decks by Thomas Vitale

Other Decks in Technology

Transcript

  1. Thomas Vitale YOW! Conference December 2024 Concerto for Java and

    AI Building Production-Ready LLM Applications @thomasvitale.com
  2. The WHY Factor What problem does it solve? How ready

    is it for production? Yeah, but how about the DevEx? @thomasvitale.com
  3. Machine Learning Subset of Arti fi cial Intelligence Platform/Infrastructure Platform

    Engineers HTTP API Application Developer Model Training Model Inference ML Engineers Data Preparation Data Scientists @thomasvitale.com
  4. Model Inference via HTTP APIs Application Model Inference Service Do

    you wanna build a snowman? HTTP Application Database Service DELETE * FROM HYPE; JDBC @thomasvitale.com
  5. Java for AI-Infused Applications Integrations with Model Inference Platform/Infrastructure Platform

    Engineers Model Training Model Inference ML Engineers Data Preparation Data Scientists Application Developers Application @thomasvitale.com
  6. Inference Services and LLMs How to choose? Managed Service Unmanaged

    Service Cloud On-Premises Proprietary Open Source @thomasvitale.com
  7. LLM Security Risks (1) OWASP Top 10 for LLM Prompt

    Injection Model Denial of Service Sensitive Information Disclosure OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/ @thomasvitale.com
  8. Semantic Search From Keywords to Meaning Application Melancholic Embedding Model

    Melancholic [42…] LIKE ‘%melancholic%' SQL Store Vector Store [42…] @thomasvitale.com
  9. Question Answering with Docs Retrieval Augmented Generation Application Melancholic instrument?

    Embedding Model Melancholic instrument? [42…] Get Similar Documents Vector Store Model Question + Similar Documents @thomasvitale.com
  10. Structured Data Extraction From Text to JSON Application Text Text

    to Structured JSON Model Database Save Structured JSON @thomasvitale.com
  11. Speech Transcription From Speech to Text Application Audio Audio to

    Text Audio Model Chat Model Text to Structured JSON @thomasvitale.com
  12. Image Processing From Image to Text Application Image Image to

    Text Image Model Chat Model Text to Structured JSON @thomasvitale.com
  13. Data Validation JSON Schema Humans in the Loop Optional Values

    Mitigating hallucination risks @thomasvitale.com
  14. LLM Security Risks (2) OWASP Top 10 for LLM Insecure

    Output Handling Excessive Agency Insecure Plugin Design OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/ @thomasvitale.com
  15. Agents Tools/Function Calling Application Is this instrument available? API Function

    Call Result Question Model Function Call Result @thomasvitale.com Function Call Request
  16. Image pack build Cloud Native Buildpacks From source code to

    container image Cloud Native Buildpacks https://buildpacks.io @thomasvitale.com
  17. Image pack build gradle bootBuildImage Cloud Native Buildpacks From source

    code to container image Cloud Native Buildpacks https://buildpacks.io @thomasvitale.com
  18. Build & Deploy Cloud Native Buildpacks Kubernetes Service Binding Native

    Executables with GraalVM Going to Production @thomasvitale.com
  19. Thomas Vitale @thomasvitale.com thomasvitale.com Concerto for Java and AI Building

    Production-Ready LLM Applications https://github.com/ThomasVitale/llm-apps-java-spring-ai https://github.com/ThomasVitale/concerto-for-java-and-ai