OpenNTF Webinars 2024: Integrating Large Language Models into Domino Applications: A Primer

Slide 1

Slide 1 text

Large Language Models in Domino Applications Serdar Basegmez Developi Information Systems, London, UK

Slide 2

Slide 2 text

Serdar Basegmez ๏ Developer/Half-blooded Admin ๏ New(ish) Londoner - Ex-Istanbulite ๏ Freelance Consultant at Developi UK ๏ Member Director at OpenNTF Board ๏ Notes/Domino since 1999 ๏ IBM Champion Alumni (2011-2018) ๏ HCL Ambassador (2020-2024) ๏ Blog: LotusNotus.com / Twitter: @serdar_basegmez ๏ Also tweets/writes/speaks/podcasts on scientific skepticism

Slide 3

Slide 3 text

Everything Open Source! Demos, source codes, libraries, integrations, datasets… https://github.com/sbasegmez

Slide 4

Slide 4 text

Today… Large Language Models Glossary of Terms Potential Applications LLM Integration Methods Assessing Our Toolbox Conclusion

Slide 5

Slide 5 text

Understanding the Impact Large Language Models: Transformative or Overhyped?

Slide 6

Slide 6 text

Game Changer? ๏ A new paradigm in programming? ๏ Programming with prompts… ๏ New ways to interact ๏ Conversation - Chat or audio ๏ Accessibility ๏ Ability to use “unusable” data ๏ Extract value from documents, audio, images ๏ Multilingual content ๏ Cultural context, specialized knowledge…

Slide 7

Slide 7 text

Or, Yet Another Big Hype? ๏ Safety, security, privacy, compliance ๏ Ethical issues ๏ Bias and Fairness ๏ “Glorified auto-complete”? ๏ Lack of creativity and critical thinking ๏ Indeterministic behaviour ๏ “Temperature” trade-off ๏ Hallucinations ๏ Scalability and Efficiency

Slide 8

Slide 8 text

Insanity Check… ๏ Nearly 80% of AI projects fail! ๏ Double rate of other IT projects. ๏ Why? ๏ Misunderstood problem definition ๏ Complex problem ๏ Data Quality and Availability ๏ Technology-driven rather than solution-focused ๏ Infrastructure is not sufficient https://www.rand.org/pubs/research_reports/RRA2680-1.html#document-details

Slide 9

Slide 9 text

Founda

Slide 10

Slide 10 text

Glossary of Terms ๏ Artificial Intelligence ๏ Machine Learning ๏ Deep Learning ๏ Natural Language Processing ๏ Generative AI ๏ Foundation Models ๏ Large Language Models Source: https://www.techtimes.com/articles/297641/20231017/deepening-our-understanding-of-artificial-intelligence-from-machine-learning-to-generative-ai-large-language-models-and-beyond.htm

Slide 11

Slide 11 text

Short History https://blogs.nvidia.com/blog/what-are-foundation-models/

Slide 12

Slide 12 text

Glossary of Terms ๏ Transformers • BERT: Bidirectional Encoder Representations from Transformers • GPT: Generative pre-trained transformer https://vinija.ai/models/Transformers/

Slide 13

Slide 13 text

Glossary of Terms - Models ๏ Large Language Models ๏ Base / Foundation Models ๏ Modalities ๏ Tasks ๏ Fine tuning https://blogs.nvidia.com/blog/what-are-foundation-models/ https://research.aimultiple.com/large-language-models/

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

What is it Good for: Large Language Model Tasks ๏ Text summarisation / Simplification ๏ Sentiment analysis ๏ Chatbots / Conversational AI ๏ Classification / Entity recognition ๏ Semantic Search ๏ Speech recognition ๏ Recommendation ๏ Text/Image/Audio/Video Generation ๏ Text-to-speech synthesis ๏ Spell/Grammar correction ๏ Translation ๏ Fraud detection ๏ Code generation ๏ AI Agents

Slide 16

Slide 16 text

Demo Suggest an OpenNTF app for logging XPages Log File Reader What ???

Slide 17

Slide 17 text

Word Embeddings Vectors and Vector Search

Slide 18

Slide 18 text

Word Embeddings ๏ Vector representation for words in multi-dimensional space https://www.cs.cmu.edu/~dst/WordEmbeddingDemo/tutorial.html

Slide 19

Slide 19 text

Word Embeddings - Real Life Vector space representation of project embeddings

Slide 20

Slide 20 text

Word Embeddings - Real Life Vector space representation of project embeddings WildFire DominoTeamMailbox XLogback XPages OpenLog Logger

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Building a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Embedding Model Project Details Project Details Project Details Project Details Project Details Vector Database embed store Ingesgng pre-process (e.g. chunking…) Project Metadata

Slide 23

Slide 23 text

Query a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Embedding Model Vector Database Matching Vectors + metadata + Scores embed Query “Logging Library” enrich query enhance Retrieval Query Transformation, Rewriting, HyDE … Reranking, Result Transformation … Recursive retrieval, Filtering … Project Metadata

Slide 24

Slide 24 text

Vector Database https://www.linkedin.com/pulse/navigating-landscape-vector-databases-comprehensive-analysis-bobbili-uuvre

Slide 25

Slide 25 text

Picking the Right Model Finding the right fit for the task

Slide 26

Slide 26 text

Model Cards https://huggingface.co/blog/model-cards

Slide 27

Slide 27 text

Word Embeddings - Models Large Language Model Local Models (e.g. Ollama, Onnx files…) ✓ Your data won’t leave the server ✓ Most are free with permissive licenses ✓ No vendor lock-in ✓ No cost per operation ! Model files are huge. ! LLM tasks are resource-intensive ! Less capable models ! Programmability restrictions

Slide 28

Slide 28 text

Word Embeddings - Models Large Language Model Cloud Models (e.g. OpenAI, Vertex AI, etc.) ✓ Managed services ✓ Pay-per-use model ✓ Easy to use - RESTful API and native SDKs ✓ Scalable / Available ✓ High performance / High quality ✓ Much better in complicated tasks ! Privacy and security concerns ! Network latency ! High costs for very busy systems ! Vendor lock-in

Slide 29

Slide 29 text

Decide and Test the Model Suggestion: Learn Python!

Slide 30

Slide 30 text

Improving Models Tweaking the Brain

Slide 31

Slide 31 text

Increase Knowledge: Fine Tune (Transfer Learning) https://www.upstage.ai/blog/tech/understanding-finetuning

Slide 32

Slide 32 text

Improve Behavior: Prompt Engineering

Slide 33

Slide 33 text

Improve Prompts: Retrieval-augmented generation ๏ Scenario • Domain Knowledge in documents, databases, etc. • LLM to respond questions aligned with domain knowledge

Slide 34

Slide 34 text

Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges

Slide 35

Slide 35 text

Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges

Slide 36

Slide 36 text

Improve Prompts: Retrieval-augmented generation Vector Database Preprocessing Chunking Embedding Indexing/Inges

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

Demo Prompts and Chat

Slide 39

Slide 39 text

Working with LLMs for Domino Apps LLM Integration is a simple REST API integration

Slide 40

Slide 40 text

Access LLMs using Java in Domino ๏ XPages ๏ OSGi Plugins ๏ RESTful API (OpenNTF JakartaEE project by Jesse) ๏ Java Agents (Notes Client or Server side) ๏ DOTS ๏ Java Addin .xsp

Slide 41

Slide 41 text

For Java Developers LangChain4j is very promising Java meets AI How to Build LLM-Powered Applications with LangChain4j

Slide 42

Slide 42 text

A New Project: Domino-LangChain4j ๏ Experimental phase ๏ Import langchain4j library into Domino ๏ Utilise ChatModel w/ Local or Cloud LLM ๏ Embedding ๏ RAG ๏ Server and Designer plugins ๏ Add some utilities ๏ Local Model Support ๏ Managed beans ๏ Configuration / Logging ๏ RAG document loaders for Domino ๏ Looking into Java Agent and DOTS support ๏ Feedbacks are welcome! Java meets AI How to Build LLM-Powered Applications with LangChain4j

Slide 43

Slide 43 text

Access LLMs using LotusScript in Domino ๏ For LotusScript, there are still options. ๏ Use RESTful access using LotusScript ๏ Use Java Agent ๏ LLM integration might be done with Java agents. LotusScript can call agents

Slide 44

Slide 44 text

Other LLM Projects ๏ HCL Domino IQ (Future Product) ๏ Uses Llama.cpp ๏ Integrated to the server ๏ Open LLM Integrator on OpenNTF ๏ Ollama integration with RAG and QDrant support ๏ By Erik Schmalz ๏ ChatGPT APIs for Domino on OpenNTF ๏ Credits: Ayhan Sahin & Christian Sadeghi

Slide 45

Slide 45 text

Integration Outside of the Domino Server ๏ Implement LLM logic in your favorite platform ๏ Volt MX ๏ Python ๏ Java ๏ JavaScript ๏ … ๏ Access to Domino Data ๏ Using Domino REST API ๏ Implement your own services with the OpenNTF Jakarta EE Project

Slide 46

Slide 46 text

Topics for Another Day… ๏ Models deep-dive ๏ Prompt Engineering ๏ Development Methodology ๏ Prototyping, validation, optimization, testing, lifecycle ๏ Safety and Security ๏ Guardrails, moderation ๏ Prompt Injections ๏ Regular Compliance Audits ๏ AI Accountability

Slide 47

Slide 47 text

Feedbacks and Discussions ๏ OpenNTF Discord Server ๏ Specific Projects —> Using LLM/AI in Domino Applications OpenNTF Discord

Slide 48

Slide 48 text

Resources ➡ All the demo materials: • https://github.com/sbasegmez/LLM-Demos ➡ OpenNTF Projects Metadata: • https://www.openntf.org/main.nsf/project.xsp?r=project/ OpenNTF Projects Dataset ➡ Domino-Langchain4j experimental version: • https://github.com/sbasegmez/domino-langchain4j

Slide 49

Slide 49 text

More Good Stuff: Odds and Ends ๏ Further reading… ๏ Huggingface blogs ๏ RAG - Retrieval Augmented Generation ๏ Multimodal approaches ๏ Prompt Engineering ๏ Courses, guides ๏ Quick Start Guide to Large Language Models (LLMs) Course by Sinan Ozdemir ๏ Large Language Models: Application through Production Databricks ๏ Large Language Model Ebooks NVidia