Building LLM-powered applications in Ruby Wroclove.rb 2024 April 14, 2024 by Andrei Bondarev

My Impact

What is Generative AI? Large Language Models (LLMs) Deep Learning Artificial Neural Networks (models) with general-purpose language understanding and generation. Exploded in popularity after the Attention Is All You Need (2017) research paper that introduced the Transformers architecture.

The GenAI Impact Supervised learning 1 month Get labeled data 3 months Train AI model on data 3 months Deploy (run) model Prompt-based AI Few days Specify prompt Few days Deploy model

LLMs excel at Structuring Data Summarizing Data Classifying Data Translating Languages Generating Content Answering Questions

LLM in every stack

AI-centric tech stack

Decision tree traversal

Business logic (in code) The Ruby on Rails promise: "Developers focus on writing business logic and not reinventing engineering solutions" Old World (before AI) Business logic in models and service objects New World (after AI) Business logic in prompts

Business logic (in prompts)

AI Agents ƻ Autonomous (semi-autonomous) general purpose LLM-powered programs Can use Tools (APIs, other systems) via 'Function Calling' Work best with powerful LLMs Can be used to automate workflows/business processes and execute multi-step tasks

Agent Reliability Responsibilities # of Tasks Decision Tree SIMPLER COMPLEX INCREASES Reliability DECREASES RELIABLE UNREALIABLE

Reliability x Focus Reliable Unreliable General Focused AGI Today Pre-Transformers Today Today

Slow adoption 1 Fast changing 2 IP / Copyright issues 3 Lack of tooling 4 Risky THE CITY - NYC News NYC AI Chatbot Touted by Adams Tells Businesses to Break the Law The Microsoft-powered bot says bosses can take workerʼs tips and that landlords can discriminate based on source of income. That's not right. aibusiness.com Air Canada Held Responsible for Chatbotʼs Hallucinations Air Canada's chatbot gave a traveler wrong airfare information. The traveler sued when the airline refused to give a refund. GM Authority GM Dealer Chat Bot Agrees To Sell 2024 Chevy Tahoe For $1 One customer recently managed to trick a dealer chat bot to agree to sell them a new 2024 Chevy Tahoe for just $1.

Prompt Engineering Alchemy Ŀ Randomness ¯\_(ツ)_/¯ [Large Language Models as Optimizers](https://arxiv.org/pdf/2309.03409.pdf)

Jailbreaking The process of getting a GenAI model to do or say unintended things through prompting Many-shot jailbreaking — including large amounts of text in a specific configuration that forces LLMs to produce potentially harmful responses, despite their being trained not to do so.

LLM Deficiencies Hallucinations Model generating incorrect or non-sensical text Outdated Data Example: GPT-4 was trained on data up to April 2023 Relevant Knowledge is Not Used

Retrieval Augmented Generation (RAG)

Detailed explanation 1 Generate vector embeddings from user's question. 2 Retrieve relevant documents by running similarity search in a vector database. 3 Construct the RAG prompt to send to the LLM. 4 Get the response back from the LLM in natural language

Vector embeddings An approach to cluster data in an N-dimensional space, organized by its meaning. LLMs encode meaning behind texts in the embedding space or "latent space". OpenAI's text-embedding-ada-002 model uses 1536 dimensions.

Vector or "Latent" space

Semantic/Vector search

RAG prompt

Naive RAG

Evals Human evals ( / ) GPT-4 RAGAS metric (0.0 - 1.0 score) Faithfulness Ensuring retrieved context can act as a justification for the generated answer Context Relevance Context is focused, with little to no irrelevant information Answer Relevance The answer addresses the actual question

Advanced RAG Strategy

Interfaces ❌ Open chat ✅ Intelligent quick actions

langchainrb ⭐ Most popular Ruby framework for building LLM-powered applications Use-cases: LLMs Agents Retrieval-Augmented Generation

Langchain.rb Approach Vendor-agnostic Support most popular providers Best-practices Staying on top of emerging research Batteries-included Ease of use

Langchain.rb Approach llm = Langchain::LLM::OpenAI.new # Cohere, Gemini, Mistral, and more llm.chat() llm.embed() Common interface for LLMs: vector_db = Langchain::LLM::Weaviate.new # Chroma, Qdrant, and more vector_db.add_texts() vector_db.similarity_search() vector_db.ask() # Naive RAG Common interface for vectorsearch DBs:

Building an Assistant

Why (not) Ruby? Pragmatism Focus on business outcomes OOP principles "There's a gem for that"

My Most Popular Tweet Twitter Andrei Bondarev on Twitter / X 1. Ruby Developer is sad a library exists in Python but not in Ruby.2. Ruby Developer copy- pastes said library' s files one by one into ChatGPT.3. ChatGPT converts Python code to Ruby code.4. Ruby Developer fixes a few undefined errors and imports the correct…

Learnings from OSS Be responsive Be friendly , Be helpful Ƽ

Langchain.rb Contributors

Langchain.rb Discord

"We do not have it all figured out yet…" "But we're optimistic!" —AI Engineering Community

Thank you! Questions? @rushing_andrei @andreibondarev in/andreibondarev [email protected]