#engageug Em03 Experimenting with Integrating Large Language Models into Domino Apps Serdar Basegmez Developi / OpenNTF

#engageug Serdar Basegmez ๏Developer/Half-blooded Admin ๏New(ish) Londoner - Ex-Istanbulite ๏Freelancer at Developi UK ๏Member Director at OpenNTF Board ๏Notes/Domino since 1999 ๏IBM Champion Alumni (2011-2018) ๏HCL Ambassador (2020-2024) ๏Blog: LotusNotus.com / Twitter: @serdar_basegmez ๏Also tweets/writes/speaks/podcasts on scientific skepticism

#engageug Today… ๏Glossary of Terms ๏Showcasing the Potential ๏Integration Scenarios ๏Assessing Our Toolbox ๏What is Next? ๏Conclusion

#engageug Impact

#engageug Understanding the Impact: Game Changer? ๏A new paradigm in programming? ๏Programming with prompts… ๏New ways to interact ๏Conversation - Chat or audio ๏Accessibility ๏Ability to use “unusable” data ๏Extract value from documents, audio, images ๏Global reach

#engageug …or is it Yet Another Big Hype? ๏Safety, security, privacy, compliance ๏Ethical issues ๏Bias and Fairness ๏“Glorified auto-complete”? ๏Lack of creativity and critical thinking ๏Indeterministic behaviour ๏“Temperature” trade-off ๏Scalability and Efficiency ๏Over-reliance

#engageug Foundations and Evolution Key concepts and their evolution

#engageug Glossary of Terms Source: https://www.techtimes.com/articles/297641/20231017/deepening-our-understanding-of-artificial-intelligence-from-machine-learning-to-generative-ai-large-language-models-and-beyond.htm ๏Artificial Intelligence ๏Machine Learning ๏Deep Learning ๏Natural Language Processing ๏Generative AI

#engageug Short History https://blogs.nvidia.com/blog/what-are-foundation-models/

#engageug Glossary of Terms ๏Transformers ๏BERT: Bidirectional Encoder Representations from Transformers ๏GPT: Generative pre-trained transformer https://vinija.ai/models/Transformers/

#engageug Glossary of Terms - Models ๏Large Language Models ๏Base / Foundation Models ๏Modalities ๏Tasks ๏Fine tuning https://blogs.nvidia.com/blog/what-are-foundation-models/ https://research.aimultiple.com/large-language-models/

#engageug Transformer Family Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond, Yang K. et a,. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond 3 2023 2022 2019 2018 2020 2021 BERT ALBERT DeBERTa ELECTRA T5 BART Flan T5 mT5 T0 GPT-1 GPT-2 XLNet OPT BLOOM GPT-Neo GPT-NeoX GPT-J LLaMA YaLM Galactica RoBERTa Open-Source ERNIE Distill BERT Switch BLOOMZ Word2Vec GloVe FastText Tk OPT-IML Encoder-Only Decoder-Only Encoder-Decoder GLM Chat GLM ST-MoE ELMo ULMFiT Fig. 1. The evolutionary tree of modern LLMs traces the development of language models in recent years and highlights some of the

#engageug What is it Good For?

#engageug Large Language Model Tasks ๏Text summarisation / Simplification ๏Sentiment analysis ๏Chatbots / Conversational AI ๏Classification / Entity recognition ๏Semantic Search ๏Speech recognition ๏Recommendation ๏Text/Image/Audio/Video Generation ๏Text-to-speech synthesis ๏Spell/Grammar correction ๏Translation ๏Fraud detection ๏Code generation ๏AI Agents

#engageug Demo

#engageug Word Embeddings Vectors and Vector Search

#engageug Word Embeddings ๏Vector representation for words in multi- dimensional space https://www.cs.cmu.edu/~dst/WordEmbeddingDemo/tutorial.html

#engageug Word Embeddings 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Large Language Model Hello World

#engageug Word Embeddings - with a Vector Store 0.39805865 0.55423045 0.28632614 -0.6990865 -0.3808561 -0.1388 0.51647455 0.6454503 0.79717076 0.43035495 0.12107085 0.3470426 -0.21693653 0.1270209 -0.81142104 0.35026655 ... ... -0.13448396 -0.10078076 0.33276576 Large Language Model Project Text Project Text Project Text Project Text Project Text Vector Database logging library query add/update store closest? Matching Vectors + metadata + Scores

#engageug Vector Database https://www.linkedin.com/pulse/navigating-landscape-vector-databases-comprehensive-analysis-bobbili-uuvre

#engageug Word Embeddings - Models Large Language Model Local Models (e.g. .onnx files) ✓ Your data won’t leave the server ✓ Most are free with permissive licenses ✓ No vendor lock-in ✓ No cost per operation 👎 Model files are huge. 👎 LLM tasks are resource-intensive 👎 Less capable models 👎 Programmability restrictions

#engageug Word Embeddings - Models Large Language Model Cloud Models (e.g. OpenAI, Vertex AI, etc.) ✓ Managed services ✓ Pay-per-use model ✓ Easy to use - RESTful API and native SDKs ✓ Scalable / Available ✓ High performance / High quality ✓ Much better in complicated tasks 👎 Privacy and security concerns 👎 Network latency 👎 High costs for very busy systems 👎 Vendor lock-in

#engageug Decide and Test the Model ๏Suggestion: Learn Python!

#engageug For Java Developers ๏LangChain4j is very promising Java meets AI How to Build LLM-Powered Applications with LangChain4j

#engageug Demo Prompts and Chat

#engageug Working with LLMs for Domino Apps LLM Integration is a simple REST API integration

#engageug Access LLMs using Java in Domino ๏XPages ๏OSGi Plugins ๏RESTful API (OpenNTF JakartaEE project by Jesse) ๏Java Agents (Notes Client or Server side) ๏DOTS ๏Java Addin .xsp

#engageug A New Project: Domino-LangChain4j ๏Experimental phase ๏Import langchain4j library into Domino ๏Utilise ChatModel w/ Local or Cloud LLM ๏Embedding ๏RAG ๏Server and Designer plugins ๏Add some utilities ๏Local Model Support ๏Managed beans ๏Configuration / Logging ๏RAG document loaders for Domino ๏Looking into Java Agent and DOTS support ๏Feedbacks are welcome! OpenNTF Discord

#engageug Bonus: Retrieval Augmented Generation https://medium.com/@tiro2000/chat-with-your-rag-97a142663d16

#engageug Access LLMs using LotusScript in Domino ๏For LotusScript, there are still options. ๏Use RESTful access using LotusScript ๏OpenNTF project for ChatGPT integration ๏Credits: Ayhan Sahin & Christian Sadeghi ๏Use Java Agent ๏LLM integration might be done with Java agents. LotusScript can call agents

#engageug Integration Outside of the Domino Server ๏Implement LLM logic in your favourite platform ๏Volt MX ๏Python ๏Java ๏JavaScript ๏… ๏Access to Domino Data ๏Using Domino REST API ๏Implement your own services with the OpenNTF Jakarta EE Project

#engageug Resources ➡ All the demo materials posted on Github: • https://github.com/sbasegmez/Engage24-Demo ➡ Domino-Langchain4j experimental version: • https://github.com/sbasegmez/domino-langchain4j

#engageug More Good Stuff: Odds and Ends ๏Further reading… ๏LangChain ๏RAG - Retrieval Augmented Generation ๏Multimodal approaches ๏Prompt Engineering ๏Courses, guides ๏Databricks - Large Language Models: Application through Production ๏NVidia - Large Language Model Ebooks ๏The Practical Guides for Large Language Models