Backend.AI Continuum을 이용한 AI Product 개발하기

Track 3_1545_Lablup Conf 2025_문현경

Transcript

2. Backend.AI Continuum을 이 용한 AI Product 개발하기 문현경, 래블업
4. What is AI:DOL? Generative AI development platform • AI:DOL(Deployable Omnimedia Lab) is Lablup's new AI-native application • Integration with Backend.AI Continuum working al ongside models powered by Backend.AI • Easy access to AI infrastructure, Making AI models s imple and accessible for all users to utilize effortlessly
5. What can AI:DOL do?
6. What is AI Native Application? Built from the ground up with AI capabilities natively • AI as a first-class citizen in every layers • Natural language and visual inputs as primary user i nput • Application is consists of • Frontend: Real-time AI feedback • APIs: Request routing to inference services • LLM: Inference & Model Serving
7. Is that Chatbot? No, AI Native Application is not just a simple Chatbot s. Modern AI product generate artifacts from requests
8. What is Artifact? In LLMs, an artifact is a piece of generated conten t created by the AI, that is presented in a way outside the main chat flow • Concrete, executable deliverables objects like docum ents, photo, video, audio, code, and more • Claude Artifacts, Google AI Studio and AI:DOL
13. How to cook AI:DOL? Here is the tech stack we used in building AI:DOL • Next.js • Frontend web app and API Routes direct communication with LLMs • AI SDK v5 • Standardized communication with LLM models • Support for streaming, token management, error handling • AI Elements • React components designed for AI applications • Provides Chat UI, Artifacts rendering, etc. • Backend.AI Continuum • Unified endpoint for multiple models. • Zero-downtime service, load balancing, traffic distribution
15. Why is chat matter in AI Applications? Chat is the best way of communicating detailed requ irements using natural language instead of technical parameters • Express Complex Ideas Simply, complex requireme nts in natural language to generate artifacts • Direct AI Access, chat provides the most intuitive w ay to test and utilize AI model capabilities • Instant Interaction, get immediate responses and it erate on ideas through conversational commands
16. How to design chat layout? We adopted AI Element Architecture
18. What should users see on AI Processing? First, let users know what AI ​​process is ongoing • Loading-indicator as soon as message sent • Streaming-state • Pulse animation • Gradient text • Chain of Thought
22. How conversations work with Backend.AI Continuum
25. What messages will users see? You can use UIMessage of AI SDK v5 to render me ssage with React. • Use `messages` in useChat hook • Represents the full application state needed for UI r endering. • `parts` property allowing for rich content like text, r easoning, tools • TextUIPart, ReasoningUIPart, SourceUrlUIPart, SourceDocu mentUIPart, FileUIPart, DataUIPart, StepStartUIPart • Custom user message
28. Streaming message is matter? Yes, Streaming is essential because they transform th e UX from "loading spinner for 30+" to "watching res ponse in real-time”. • Responses should appear natural to the user • Tune AISDK options for streaming • StreamTextTransform(line, word) • experimental_throttle • Ready for rendering incomplete markdown • Multiple rendering happened in code highlighting • Streamdown, AI Elements powered
31. How to generate artifacts?
33. Why should we make two streams? Main streams keeps the conversation flowing with t he user, while tool stream handles tool results • Main LLM Stream (User-Facing) • User communication only • Deciding when to call tools, description of tool usage • Tool Stream Responsibilities • Executing the tool properly (with other models) • Managing separate data streams, onData • Use clear prompts for both models • “A document has been created and is now visible to the user”
34. How to manage data-events
36. How about code generation? We have a build and code-sandbox for code generati on • Transpiling and executing JavaScript /TypeScript cod e generated by AI models • Monaco Editor and `importmap` • Running JS code in Node.js environment and Pytho n with external packages • CodeContainer (WebContainer and pyodide) • Updating codes directly on the AI:DOL
38. Can you generate images? Image generation is awesome, you can also generat e images via prompts • Image Model allows us to generate images using pr ompt • OpenAI Dall-E, Google Gemini 2.5 Flash Image, and more • Saving blob image data on the server, and show it • Consider image rendering with partial images
40. What is in the next? Multimodal supporting Support more generative content, ex) Video, Audio Tool-calling with Human in the loop MCP in sandbox Code-sandbox on the sever-side Context compacting Quota management Workflow pipelines Working and testing with more opened models