Beyond One LLM: Tracking Knowledge Changes with Neo4j

Presented at NODES 2026 (November 12, 2026).

LLMs, prompts, and data sources are constantly evolving. How can we track changes in knowledge, compare different perspectives, and preserve what we have learned?

This session introduces Knowledge Graph Diff, an approach to managing evolving knowledge in Neo4j using concepts inspired by Git, such as snapshots, diffs, validation, and merging.

Using practical examples, it explores how Neo4j can track claims, evidence, and relationships across multiple LLMs and RAG workflows while preserving provenance and traceability.

The goal is not to find one perfect answer, but to build a reusable knowledge layer that evolves beyond individual LLMs.

Official session page:
https://neo4j.com/nodes/agenda/beyond-one-llm-tracking-knowledge-changes-with-neo4j/

Transcript

1. NODES 2026 — Thursday, 12 November 2026 — 12:30–13:00 JST (UTC+9) Beyond One LLM: Tracking Knowledge Changes with Neo4j 1 Koji Annoura Neo4j Ninja @kojiannoura koji@annoura.com Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
2. Who Am I? ❖ Koji Annoura ❖ Fukuoka, Japan ❖ Neo4j Ninja ❖ Co-founder of Neo4j Users Group Tokyo ❖ Graph Community MVP Asia 2020 ❖ Building Practical Knowledge Graphs ❖ Cypher, GQL, SQL/PGQ 2
3. Today’s Journey ❖ 1. Why Knowledge Changes ❖ 2. Knowledge Graph Diff with Neo4j ❖ 3. Real-World Example: Node.js and LLMs 3
4. 1. Why Knowledge Changes 4
5. Software Changes. Knowledge Changes. ❖ Software keeps changing ❖ New releases and dependencies ❖ Security vulnerabilities ❖ Knowledge also changes ❖ New evidence and assessments ❖ New decisions ❖ How can we track both? 5 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
6. What Happens When We Change LLMs? ❖ LLMs and prompts keep changing ❖ Different models may give different answers ❖ Important context may be lost ❖ Evidence and history ❖ Decisions and their reasons ❖ Knowledge should survive model changes 6 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
7. 2. Knowledge Graph Diff with Neo4j 7
8. Beyond GraphRAG and Agent Memory ❖ GraphRAG retrieves knowledge ❖ Agent Memory preserves context ❖ Knowledge Graph Diff ❖ Compares knowledge ❖ Tracks changes ❖ Preserves history and decisions 8 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
9. What Is Knowledge Graph Diff? ❖ Treat knowledge as a versioned asset ❖ Compare across ❖ Time and sources ❖ LLMs and prompts ❖ Understand what changed and why ❖ Preserve alternative interpretations 9 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
10. Inspired by Git ❖ Snapshot — Capture a knowledge state ❖ Diff — Identify changes ❖ Branch — Preserve alternatives ❖ Validation — Check claims and evidence ❖ Merge — Integrate validated knowledge 10 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
11. Three Levels of Diff ❖ Structural Diff ❖ Packages, versions, dependencies ❖ Evidence Diff ❖ Advisories, release notes ❖ Knowledge Diff ❖ Claims, assessments, decisions 11 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
12. From SPDX to a Knowledge Graph ❖ SPDX describes software components ❖ Packages and versions ❖ Dependency relationships ❖ Import SBOM snapshots into Neo4j ❖ Preserve original identi ers and sources fi 12 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
13. Connecting Evidence and Decisions ❖ Software ❖ Components and dependencies ❖ Evidence ❖ Advisories, release notes, commits ❖ Knowledge ❖ Claims and assessments ❖ Decisions ❖ Alternatives, rationale, actions 13 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
14. SEPT: Sharing Knowledge, Preserving Different Viewpoints ❖ SEPT ❖ S stands for Shared — knowledge others can understand and reuse. ❖ E stands for Evolving — knowledge can change with new evidence. ❖ P stands for Plural — we preserve different interpretations. ❖ T stands for Traceable — we connect knowledge to sources, evidence, and decisions. ❖ These principles apply to both humans and AI. 14 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
15. 3. Real-World Example: Node.js and LLMs 15
16. Demo: A Real-World Web Application ❖ An Astro application built with Node.js ❖ Real development history ❖ Git commits ❖ Release notes ❖ Capture three software snapshots ❖ package.json and package-lock.json ❖ Node.js runtime information ❖ Store connected information in Neo4j 16 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
17. Starting with LLM A ❖ LLM A analyzes the application ❖ Software snapshots and dependencies ❖ Available security information ❖ Extract claims and assessments ❖ Preserve sources and provenance ❖ Create Knowledge Snapshot A 17 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
18. A New Security Vulnerability ❖ A Node.js vulnerability is disclosed ❖ Security advisory ❖ Affected versions ❖ New evidence becomes available ❖ The software has not changed yet ❖ Our knowledge has changed 18 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
19. Responding to the Vulnerability ❖ Review the security advisory ❖ Update the affected Node.js version ❖ Record the action ❖ Git commit ❖ Release note ❖ Capture a new software snapshot 19 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
20. Comparing Software Changes ❖ Compare software snapshots using Cypher ❖ Identify ❖ Version changes ❖ Declared, resolved, and runtime versions ❖ Trace changes to commits and evidence ❖ Visualize the structural diff 20 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
21. Software Snapshots in Neo4j 21 Source: Original demonstration data visualized with Neo4j Browser, Koji Annoura, 2026.
22. Structural Diff: Declared, Resolved, and Runtime 22 Source: Original Cypher queries and demonstration data, Koji Annoura, 2026.
23. Comparing Two LLMs with Shared Knowledge ❖ Compare ChatGPT and Claude ❖ Use the same evidence ❖ Node.js release notes ❖ Application context ❖ Store and compare claims in Neo4j 23 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
24. Two LLMs, Same Decision Without Release Notes Comparison ChatGPT Claude Node.js versions v26.12.1 → v26.13.0 Same Release notes Not provided Not provided Recommendation WAIT WAIT Con dence Low Low to moderate Reading of version no. Same major version Minor update Staging tests Noted as unknown Suggested as a next step Next step Review the evidence Review the evidence fi 24
25. A New Node.js LTS Arrives ❖ Two valid options ❖ Keep the current version ❖ Upgrade to the new LTS ❖ Why should we upgrade again? 25 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
26. Recommendation Diff: Before and After 26 Source: Original Cypher queries and demonstration data, Koji Annoura, 2026.
27. Shared and Model-Specific Claims 27 Source: Original Cypher queries and demonstration data, Koji Annoura, 2026.
28. Tracing Claims to Evidence in Neo4j 28 Source: Original Cypher queries and demonstration data, Koji Annoura, 2026.
29. Same Evidence, Different Decisions Topic ChatGPT Claude LTS status Con rmed Con rmed Release changes No additional code changes No additional code changes Application validation Not supplied Not supplied Upgrade assessment Validation needed Low risk (inferred) Recommendation WAIT UPGRADE fi fi 29
30. Tracking Decisions with Neo4j ❖ Evidence ❖ Node.js v26.13.0 became LTS ❖ Of cial release and support information ❖ Alternatives ❖ Stay on v26.12.1 ❖ Upgrade to v26.13.0 ❖ Decision ❖ Upgrade to v26.13.0 ❖ Rationale ❖ Prefer long-term support and maintenance ❖ Traceability ❖ Connect decision, evidence, and Git commit fi 30 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
31. Tracking Decisions and Evidence fi 31 Source: Koji Annoura, 2026. Based on of cial Node.js release notes.
32. One Knowledge Layer, Many LLMs ❖ Cloud and local LLMs can be replaced ❖ Neo4j preserves ❖ Claims and evidence ❖ Decisions and rationale ❖ Versions and history ❖ New LLMs reuse existing knowledge ❖ No dependency on one model 32 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
33. Knowledge Is More Than a Diff ❖ A diff shows what changed ❖ Evidence supports claims ❖ Decisions explain actions ❖ History connects knowledge over time ❖ Knowledge continues to evolve 33 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
34. Should We Upgrade Neo4j? ❖ Storage Changes ❖ New store formats ❖ Migration requirements ❖ Cypher Changes ❖ New features and syntax ❖ Deprecated or removed features ❖ Compatibility ❖ Drivers, plugins, applications ❖ Decision ❖ Upgrade now or wait? ❖ What needs to change? 34 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
35. Knowledge Should Outlive LLMs ❖ Preserve knowledge beyond individual models ❖ Track software, evidence, and decisions ❖ Keep the reasons behind changes ❖ Understand not only what changed, but why 35 Source: Conceptual illustration prepared with OpenAI tools for Koji Annoura, 2026.
36. Thank You ❖ Thanks for listening! ❖ Koji Annoura ❖ https://annoura.com ❖ koji@annoura.com ❖ @kojiannoura 36