RAG: Accuracy and Explainability in GenAI Applications Jennifer Reif [email protected] @JMHReif github.com/JMHReif jmhreif.com linkedin.com/in/jmhreif 

Who is Jennifer Reif? Developer Advocate, Neo4j • Continuous learner • Technical speaker • Tech blogger, podcaster • Other: geek Jennifer Reif [email protected] @JMHReif github.com/JMHReif jmhreif.com linkedin.com/in/jmhreif 

Negative AI stories Even well-respected companies get it wrong • Hallucinating non-existent policy, legal cases • Chatbot produces Python • Legally binding vehicle o ff er • Harmful health advice • Threatening users • Inventing new language • Illegal activities (insider trading + local health laws) 

Throw an LLM at it Doesn’t often work • LLM strengths: • producing human-consumable output • probabilistic answers • LLM weaknesses: • complex logic • recognizing nuances • evaluating/expressing sensitive topics 

Demo - chatgpt.com 

How do hallucinations happen? LLM limitations • Lacking recent data • Too broad / general search • Not enough context • Uncertainty 

Adding context to the LLM What does it do? • Guides it • Focuses / narrows search area • Adds to LLM knowledge • Reduces margin of error 

Demo - chatgpt.com 

How to provide context …that we already have! • Plug and play with existing data • High-quality data • Internal value • Information in new ways 

RAG architecture • Retrieval • Data retrieved from external source • Augmented • Augments response with facts • Generation • Response in natural language Prompt + Relevant Information LLM API LLM
 Chat API User Database Search Prompt Response Relevant Results / Documents 2 3 1 Database 

Retrieval source options • Vector database • Relational (+ vectors) • NoSQL (+ vectors) • Graph (+ vectors) • Other: Directories / Websites / etc 

Our demo Book recommendations • Search terms / topics • Book descrs -> User reviews • Lexical search -> semantic search • Less needle-in-a-haystack • More relevant! 

Vector dbs 

Embeddings / Vectors Convert data to a point in space • Series of numbers • 100s or 1000s of dimensions • Dimension = interesting feature / characteristic 

Vectors in the physical realm https://www.mathsisfun.com/algebra/vectors.html 

Vectors in the technical realm Kings and Queens king − man + woman ≈ queen king man wom an 1 king man wom an 2 queen? 3 

How do we search the vectors? Similarity search • Expensive queries (compare to every vector) • Approximate nearest neighbor (k-ANN) • Example: Library • Book classi fi cation - genre vs location of plot • Smaller search set = smaller retrieval time! Photo by Martin Adams on Unsplash 

Pinecone: sample data Document{ id='10609bf6-b358-449e-a4d1-8a6f2a2f805d', text='As always a Page turner. may me think about where we are in our tech evolution.', media='null', metadata={ rating=4.0, book_id=18505765, distance=0.58059925 }, score=0.41940075159072876 } 

Where do vectors fall flat? How do you… • Verify vector representations? • Explain how it got to answer? • Ensure highest relevance / accuracy? 

Graph dbs 

What is a graph? Answers through relationships • Coworkers shared classes/degrees? • Common degree journeys? • Alumni re-enroll for higher degrees? • Who else went to my school? Person Degree Degree Company Person Person School ATTENDED ATTENDED W ORKED_FOR WORKED_FOR EN RO LLED_IN ENROLLED_IN ENROLLED_IN Degree C O M PLETED COM PLETED Edward Jones Jacob Jennifer SIUE Music CMIS CS Adrian 

Graphs add context and meaning Not just point-to-point, but HOW they connect 

What does it solve? Data problems • Documented path (not just data) • Answering how and why • Understanding/ fi nd hidden connections • Find alternates, impacts, etc. 

Neo4j: sample data Book[ id=18505765, title=Dark Matter (Star Carrier, #5), isbn=0062183990, isbn13=9780062183996, average_rating=3.93, authors=[ Author[name=Ian Douglas] ], reviewList=[ Review[ id=f78b825c122cc5b924957fafc7382bc1, text=As always a Page turner. may me think about where we are in our tech evolution., book_id=18505765, rating=4 ] ] ] 

Vector vs Graph RAG: 

How are they similar? • Store and index data for e ffi cient retrieval • Query data with queries • Semantic searches • Retrieve metadata / related data 

Index differences Vector db vs Graph db • Top unit of vector data • Data upserted into index • Del index = del data • Search vs Semantic indexes • Use explicit proc in Cypher • Del index != del data https://glennas.wordpress.com/2011/03/13/understanding-graph-databases-marko-rodriguez/ https://docs.pinecone.io/guides/indexes/understanding-indexes 

Storage practices Vector db vs Graph db • Prioritize stats • Some metadata / connections • Avoid large data types / values • Prioritize relationships • Value = metadata / connections • Avoid large types as properties Document{ id='10609bf6-b358-449e-a4d1-8a6f2a2f805d', text='As always a Page turner. may me think about where we are in our tech evolution.', media='null', metadata={ rating=4.0, book_id=18505765, distance=0.58059925 }, score=0.41940075159072876 } Book[ id=18505765, title=Dark Matter (Star Carrier, #5), isbn=0062183990, isbn13=9780062183996, average_rating=3.93, authors=[ Author[name=Ian Douglas] ], reviewList=[ Review[ id=f78b825c122cc5b924957fafc7382bc1, text=As always a Page turner. may me think about where we are in our tech evolution., book_id=18505765, rating=4 ] ] ] 

Graphs = extra layer • Accuracy: • extra context / related info in connections • Veri fi ability: • check against understandable format • Explainability: • trace path through graph for answer 

Demo! 

Resources • Github repository (today’s code): github.com/JMHReif/vector-graph-rag • Docs for Spring AI: https://docs.spring.io/spring-ai/reference/api/vectordbs.html • GraphAcademy LLM courses: graphacademy.neo4j.com/categories/llms/ • Knowledge graph ebook: https://neo4j.com/whitepapers/developers-guide-how- to-build-knowledge-graph/ Jennifer Reif [email protected] @JMHReif github.com/JMHReif jmhreif.com linkedin.com/in/jmhreif