OpenNTF Webinars 2024: Integrating Large Language Models into Domino Applications: A Primer

Integrating Large Language Models into Domino Applications: A Primer

With the AI hype at its peak, many Domino customers are sitting on vast silos of unstructured text data, waiting to be put to use. Domino applications could offer fantastic use cases —but where do we start? Join us to explore how Large Language Models (LLMs) can be integrated directly into Domino applications. We’ll begin with the basics of LLMs and move into practical steps for embedding common LLM tasks across various scenarios. Perfect for developers looking to expand Domino’s capabilities and bring the power of LLMs into their workflows.

Serdar Basegmez is an HCL Ambassador (2020-2024) and a former IBM Champion (2011-2018). As a consultant at Developi in the UK, he brings over two decades of expertise in Notes/Domino development and administration. A regular speaker at numerous conferences, Serdar also serves as a Member Director on the OpenNTF Board and is an active contributor to multiple open-source projects.

OpenNTF Webinars - 21 November 2024
https://www.youtube.com/watch?v=TtdR1sX1Kbg

Transcript

1. Large Language Models in Domino Applications Serdar Basegmez Developi Information Systems, London, UK
2. Serdar Basegmez ๏ Developer/Half-blooded Admin ๏ New(ish) Londoner - Ex-Istanbulite ๏ Freelance Consultant at Developi UK ๏ Member Director at OpenNTF Board ๏ Notes/Domino since 1999 ๏ IBM Champion Alumni (2011-2018) ๏ HCL Ambassador (2020-2024) ๏ Blog: LotusNotus.com / Twitter: @serdar_basegmez ๏ Also tweets/writes/speaks/podcasts on scientific skepticism
3. Everything Open Source! Demos, source codes, libraries, integrations, datasets… https://github.com/sbasegmez
4. Today… Large Language Models Glossary of Terms Potential Applications LLM Integration Methods Assessing Our Toolbox Conclusion
5. Understanding the Impact Large Language Models: Transformative or Overhyped?
6. Game Changer? ๏ A new paradigm in programming? ๏ Programming with prompts… ๏ New ways to interact ๏ Conversation - Chat or audio ๏ Accessibility ๏ Ability to use “unusable” data ๏ Extract value from documents, audio, images ๏ Multilingual content ๏ Cultural context, specialized knowledge…
7. Or, Yet Another Big Hype? ๏ Safety, security, privacy, compliance ๏ Ethical issues ๏ Bias and Fairness ๏ “Glorified auto-complete”? ๏ Lack of creativity and critical thinking ๏ Indeterministic behaviour ๏ “Temperature” trade-off ๏ Hallucinations ๏ Scalability and Efficiency
8. Insanity Check… ๏ Nearly 80% of AI projects fail! ๏ Double rate of other IT projects. ๏ Why? ๏ Misunderstood problem definition ๏ Complex problem ๏ Data Quality and Availability ๏ Technology-driven rather than solution-focused ๏ Infrastructure is not sufficient https://www.rand.org/pubs/research_reports/RRA2680-1.html#document-details
9. Founda<ons and Evolu<on Key concepts and their progression
10. Glossary of Terms ๏ Artificial Intelligence ๏ Machine Learning ๏ Deep Learning ๏ Natural Language Processing ๏ Generative AI ๏ Foundation Models ๏ Large Language Models Source: https://www.techtimes.com/articles/297641/20231017/deepening-our-understanding-of-artificial-intelligence-from-machine-learning-to-generative-ai-large-language-models-and-beyond.htm
11. Short History https://blogs.nvidia.com/blog/what-are-foundation-models/
12. Glossary of Terms ๏ Transformers • BERT: Bidirectional Encoder Representations from Transformers • GPT: Generative pre-trained transformer https://vinija.ai/models/Transformers/
13. Glossary of Terms - Models ๏ Large Language Models ๏ Base / Foundation Models ๏ Modalities ๏ Tasks ๏ Fine tuning https://blogs.nvidia.com/blog/what-are-foundation-models/ https://research.aimultiple.com/large-language-models/
15. What is it Good for: Large Language Model Tasks ๏ Text summarisation / Simplification ๏ Sentiment analysis ๏ Chatbots / Conversational AI ๏ Classification / Entity recognition ๏ Semantic Search ๏ Speech recognition ๏ Recommendation ๏ Text/Image/Audio/Video Generation ๏ Text-to-speech synthesis ๏ Spell/Grammar correction ๏ Translation ๏ Fraud detection ๏ Code generation ๏ AI Agents
16. Demo Suggest an OpenNTF app for logging XPages Log File Reader What ???
17. Word Embeddings Vectors and Vector Search
18. Word Embeddings ๏ Vector representation for words in multi-dimensional space https://www.cs.cmu.edu/~dst/WordEmbeddingDemo/tutorial.html
19. Word Embeddings - Real Life Vector space representation of project embeddings
20. Word Embeddings - Real Life Vector space representation of project embeddings WildFire DominoTeamMailbox XLogback XPages OpenLog Logger
21. Building a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Embedding Model Project Details Embedding
22. Building a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Embedding Model Project Details Project Details Project Details Project Details Project Details Vector Database embed store Ingesgng pre-process (e.g. chunking…) Project Metadata
23. Query a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Embedding Model Vector Database Matching Vectors + metadata + Scores embed Query “Logging Library” enrich query enhance Retrieval Query Transformation, Rewriting, HyDE … Reranking, Result Transformation … Recursive retrieval, Filtering … Project Metadata
24. Vector Database https://www.linkedin.com/pulse/navigating-landscape-vector-databases-comprehensive-analysis-bobbili-uuvre
25. Picking the Right Model Finding the right fit for the task
26. Model Cards https://huggingface.co/blog/model-cards
27. Word Embeddings - Models Large Language Model Local Models (e.g. Ollama, Onnx files…) ✓ Your data won’t leave the server ✓ Most are free with permissive licenses ✓ No vendor lock-in ✓ No cost per operation ! Model files are huge. ! LLM tasks are resource-intensive ! Less capable models ! Programmability restrictions
28. Word Embeddings - Models Large Language Model Cloud Models (e.g. OpenAI, Vertex AI, etc.) ✓ Managed services ✓ Pay-per-use model ✓ Easy to use - RESTful API and native SDKs ✓ Scalable / Available ✓ High performance / High quality ✓ Much better in complicated tasks ! Privacy and security concerns ! Network latency ! High costs for very busy systems ! Vendor lock-in
29. Decide and Test the Model Suggestion: Learn Python!
30. Improving Models Tweaking the Brain
31. Increase Knowledge: Fine Tune (Transfer Learning) https://www.upstage.ai/blog/tech/understanding-finetuning
32. Improve Behavior: Prompt Engineering
33. Improve Prompts: Retrieval-augmented generation ๏ Scenario • Domain Knowledge in documents, databases, etc. • LLM to respond questions aligned with domain knowledge
34. Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges<ng Documents
35. Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges<ng Documents Opgm ize and Vectorize Retrieve & Opgmize Augmented Query Relevant Context Relevant Context Relevant Context Retrieved Context Chat History Large Language Model Prompt Ques<on Ques<on
36. Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges<ng Documents Opgm ize and Vectorize Retrieve & Opgmize Augmented Query Relevant Context Relevant Context Relevant Context Retrieved Context Chat History Large Language Model Prompt Ques<on Ques<on Pre-Retrieval Opgmizagons Post-Retrieval Opgmizagons
38. Demo Prompts and Chat
39. Working with LLMs for Domino Apps LLM Integration is a simple REST API integration
40. Access LLMs using Java in Domino ๏ XPages ๏ OSGi Plugins ๏ RESTful API (OpenNTF JakartaEE project by Jesse) ๏ Java Agents (Notes Client or Server side) ๏ DOTS ๏ Java Addin .xsp
41. For Java Developers LangChain4j is very promising Java meets AI How to Build LLM-Powered Applications with LangChain4j
42. A New Project: Domino-LangChain4j ๏ Experimental phase ๏ Import langchain4j library into Domino ๏ Utilise ChatModel w/ Local or Cloud LLM ๏ Embedding ๏ RAG ๏ Server and Designer plugins ๏ Add some utilities ๏ Local Model Support ๏ Managed beans ๏ Configuration / Logging ๏ RAG document loaders for Domino ๏ Looking into Java Agent and DOTS support ๏ Feedbacks are welcome! Java meets AI How to Build LLM-Powered Applications with LangChain4j
43. Access LLMs using LotusScript in Domino ๏ For LotusScript, there are still options. ๏ Use RESTful access using LotusScript ๏ Use Java Agent ๏ LLM integration might be done with Java agents. LotusScript can call agents
44. Other LLM Projects ๏ HCL Domino IQ (Future Product) ๏ Uses Llama.cpp ๏ Integrated to the server ๏ Open LLM Integrator on OpenNTF ๏ Ollama integration with RAG and QDrant support ๏ By Erik Schmalz ๏ ChatGPT APIs for Domino on OpenNTF ๏ Credits: Ayhan Sahin & Christian Sadeghi
45. Integration Outside of the Domino Server ๏ Implement LLM logic in your favorite platform ๏ Volt MX ๏ Python ๏ Java ๏ JavaScript ๏ … ๏ Access to Domino Data ๏ Using Domino REST API ๏ Implement your own services with the OpenNTF Jakarta EE Project
46. Topics for Another Day… ๏ Models deep-dive ๏ Prompt Engineering ๏ Development Methodology ๏ Prototyping, validation, optimization, testing, lifecycle ๏ Safety and Security ๏ Guardrails, moderation ๏ Prompt Injections ๏ Regular Compliance Audits ๏ AI Accountability
47. Feedbacks and Discussions ๏ OpenNTF Discord Server ๏ Specific Projects —> Using LLM/AI in Domino Applications OpenNTF Discord
48. Resources ➡ All the demo materials: • https://github.com/sbasegmez/LLM-Demos ➡ OpenNTF Projects Metadata: • https://www.openntf.org/main.nsf/project.xsp?r=project/ OpenNTF Projects Dataset ➡ Domino-Langchain4j experimental version: • https://github.com/sbasegmez/domino-langchain4j
49. More Good Stuff: Odds and Ends ๏ Further reading… ๏ Huggingface blogs ๏ RAG - Retrieval Augmented Generation ๏ Multimodal approaches ๏ Prompt Engineering ๏ Courses, guides ๏ Quick Start Guide to Large Language Models (LLMs) Course by Sinan Ozdemir ๏ Large Language Models: Application through Production Databricks ๏ Large Language Model Ebooks NVidia