How I built a second brain using Firebase, Gemini, Genkit, and Siri

In our fast paced world, there is just too much information, and it often seems impossible to keep up with everything that’s going on.

If you ever felt that you couldn’t possibly remember everything you saw, read, or even didn’t read, come to this talk and I will show you how I build an app that allows me to do just that.

I will show you how I

* used SwiftUI to build a beautiful app that works across Apple’s platforms
* used Cloud Firestore to store gigabytes of data, keeping it in sync across all of my devices
* used Gemini, Google’s LLM, to summarise articles, and ask my app questions about articles
* used Genkit - an AI integration framework - to connect Gemini to my personal data store
* used Siri to provide a natural language interface that allows me to query my knowledge base hands-free

Transcript

1. Peter Friese Developer Advocate, Firebase @peterfriese How I built a second brain Using Firebase, Gemini, Genkit, and Siri
2. @peterfriese Developer Advocate, Firebase Peter Friese peterfriese.dev @peterfriese youtube.com/c/PeterFriese/ not-only-swift.peterfriese.dev/ @peterfriese@iosdev.space youtube.com/c/Firebase
3. Making Siri sma rt er…
4. Information overload The problem
5. Information overload
6. “There’s an app for that”
7. Not invented here
13. Read it later The solution
14. Codename Sofia ✨Add links via iOS Share Sheet ✨Extract OpenGraph Metadata ✨Simplified formatting ✨Offline reading
15. “I still need to read all those articles!” The challenge
16. Building an AI summariser The solution
17. https: // ai.google.dev/
18. https: // aistudio.google.com/
19. https: // aistudio.google.com/
20. Google AI SDK ✨Access Gemini models ✨SDKs for Python, Node, Go, Kotlin, Dart, Swift ✨Use for prototyping ✨Relatively low RPM ✨Requires an API key https: / / github.com/google-gemini/generative-ai-swift
21. Vertex AI in Firebase SDK ✨Access Gemini models ✨SDKs for Swift, Kotlin, JS, Dart ✨Part of Firebase ✨Enterprise-ready ✨No API key required ✨Supports App Check https: / / firebase.google.com/docs/vertex-ai
22. Using the VertexAI in Firebase SDK public func summarise(text: String) async throws -> String { let prompt = "Please summarise the following text for me: \(text)" let flash = "gemini-1.5-flash" let model = VertexAI.vertexAI().generativeModel(modelName: flash) let response = try await model.generateContent(prompt) if let text = response.text { return text } else { return "Couldn't compute summary" } } async, as we will be calling a remote system
23. Using the VertexAI in Firebase SDK public func summarise(text: String) async throws -> String { let prompt = "Please summarise the following text for me: \(text)" let flash = "gemini-1.5-flash" let model = VertexAI.vertexAI().generativeModel(modelName: flash) let response = try await model.generateContent(prompt) if let text = response.text { return text } else { return "Couldn't compute summary" } }
24. Using the VertexAI in Firebase SDK public func summarise(text: String) async throws -> String { let prompt = "Please summarise the following text for me: \(text)" let flash = "gemini-1.5-flash" let model = VertexAI.vertexAI().generativeModel(modelName: flash) let response = try await model.generateContent(prompt) if let text = response.text { return text } else { return "Couldn't compute summary" } } here’s the remote system call
25. Using the VertexAI in Firebase SDK public func summarise(text: String) async throws -> String { let prompt = "Please summarise the following text for me: \(text)" let flash = "gemini-1.5-flash" let model = VertexAI.vertexAI().generativeModel(modelName: flash) let response = try await model.generateContent(prompt) if let text = response.text { return text } else { return "Couldn't compute summary" } }
26. Demo
27. Conclusion ✨Use AI Studio to prototype your prompts ✨Use Google AI SDK to call LLMs from your client ✨Use Vertex AI in Firebase SDK to go to production ✨Secure your app with App Check ✨Call LLM directly from the client
28. Chatting with my second brain Going one step further
29. 01 Vector Embeddings 02 RAG Retrieval Augmented Generation
30. Vector Embeddings Task: Find all words that are food in the following sentence
31. Task: Find all words that are food in the following sentence “I went down to Aberystwyth on foot to buy some welsh cakes and a few berries. When I finished doing my groceries, I had a latte at Coffee #1, where I met a few other speakers.”
32. Task: Find all words that are food in the following sentence “I went down to Aberystwyth on foot to buy some welsh cakes and a few berries. When I finished doing my groceries, I had a latte at Coffee #1, where I met a few other speakers.”
33. Task: Find all words that are food in the following sentence “I went down to Aberystwyth on foot to buy some welsh cakes and a few berries. When I finished doing my groceries, I had a latte at Coffee #1, where I met a few other speakers.”
34. Vector embedding: A numerical representation of a word, sentence, or any other unit of text.
35. Vector embedding for food: [-0.018035058, 0.013980114, -0.01309541, 0.024956783, 0.02708295, -0.074924484, 0.03496225, 0.0125780115, . .. ]
36. Vector embedding for food: [-0.018035058, 0.013980114, -0.01309541, 0.024956783, 0.02708295, -0.074924484, 0.03496225, 0.0125780115, . .. ] Vector embedding for foot: [-0.016025933, 0.008207399, -0.03572462, 0.020942606, -0.0003162824, -0.041694388, 0.050102886, 0.007380137, . .. ]
37. [51.50721, -0.12758] Coordinates for London
38. Paris [48.85661, 2.35222] London [51.50721, -0.12758] New York [40.71277, -74.00597] Boston [42.36008, -71.05888]
39. Paris [48.85661, 2.35222] London [51.50721, -0.12758] New York [40.71277, -74.00597] Boston [42.36008, -71.05888]
40. Source: https://www.learndatasci.com/glossary/cosine-similarity/ Cosine vector similarity
42. 01 Serverless (There are servers, but Google manages them for you) 02 03 04 Automatic scaling Scale down to zero. Support for min/max instances. Trusted environment JavaScript / TypeScript / Python
43. Data wri tt en New user created User signing in Image uploaded Crashlytics ale r Analytics Conversions File deleted Data deleted Test run completed Data updated New con fi guration Con fi guration rollback Triggers
45. Computing Vector Embeddings export const updateEmbeddings = onDocumentWritten( { document: 'artifacts/{documentId}', minInstances: 1 }, async (event) => { const newData = event.data ?. after.data() ?? {}; const previousData = event.data ?. before.data() ? ? {}; if (newData.fullText === previousData.fullText) { return; } const embeddings = await embed({ embedder: textEmbeddingGecko, content: Document.fromText(newData.fullText), }); await event.data ?. after.ref.update({ embeddings: FieldValue.vector(embeddings), }); }); Firestore trigger
46. Computing Vector Embeddings export const updateEmbeddings = onDocumentWritten( { document: 'artifacts/{documentId}', minInstances: 1 }, async (event) => { const newData = event.data ?. after.data() ?? {}; const previousData = event.data ?. before.data() ? ? {}; if (newData.fullText === previousData.fullText) { return; } const embeddings = await embed({ embedder: textEmbeddingGecko, content: Document.fromText(newData.fullText), }); await event.data ?. after.ref.update({ embeddings: FieldValue.vector(embeddings), }); }); Read data from document
47. Computing Vector Embeddings export const updateEmbeddings = onDocumentWritten( { document: 'artifacts/{documentId}', minInstances: 1 }, async (event) => { const newData = event.data ?. after.data() ?? {}; const previousData = event.data ?. before.data() ? ? {}; if (newData.fullText === previousData.fullText) { return; } const embeddings = await embed({ embedder: textEmbeddingGecko, content: Document.fromText(newData.fullText), }); await event.data ?. after.ref.update({ embeddings: FieldValue.vector(embeddings), }); }); Compute embeddings
48. Computing Vector Embeddings export const updateEmbeddings = onDocumentWritten( { document: 'artifacts/{documentId}', minInstances: 1 }, async (event) => { const newData = event.data ?. after.data() ?? {}; const previousData = event.data ?. before.data() ? ? {}; if (newData.fullText === previousData.fullText) { return; } const embeddings = await embed({ embedder: textEmbeddingGecko, content: Document.fromText(newData.fullText), }); await event.data ?. after.ref.update({ embeddings: FieldValue.vector(embeddings), }); }); Update Firestore document
49. 01 Compute embeddings for user’s query Using embedding model 02 03 04 Find nearest neighbors Using Vector Similarity Search Inject context / history into prompt Using result of vector search / chat history Generate answer Using LLM Retrieval Augmented Generation
50. const messageSchema = z.object({ message: z.string(), role: z.enum(['user', 'system']), }); export const inputSchema = z.object({ question: z.string(), history: z.array(messageSchema).optional(), }); export const semanticQAFlow = onFlow( { name: 'semanticQAFlow', httpsOptions: { minInstances: 1 }, inputSchema: inputSchema, outputSchema: z.string(), authPolicy: firebaseAuth((user) = > { }), Genkit AI flow RAG-powered Q&A with Genkit
51. const messageSchema = z.object({ message: z.string(), role: z.enum(['user', 'system']), }); export const inputSchema = z.object({ question: z.string(), history: z.array(messageSchema).optional(), }); export const semanticQAFlow = onFlow( { name: 'semanticQAFlow', httpsOptions: { minInstances: 1 }, inputSchema: inputSchema, outputSchema: z.string(), authPolicy: firebaseAuth((user) = > { }), Format of the data we will pass in RAG-powered Q&A with Genkit
52. export const inputSchema = z.object({ question: z.string(), history: z.array(messageSchema).optional(), }); export const semanticQAFlow = onFlow( { name: 'semanticQAFlow', httpsOptions: { minInstances: 1 }, inputSchema: inputSchema, outputSchema: z.string(), authPolicy: firebaseAuth((user) = > { }), }, async (input) => { const docs = await retrieve({ retriever: firestoreRetriever, query: input.question, options: {limit: 3}, }); const context = docs.map((doc) => doc.text()).join('\n\n'); const history = input.history ? . map((message) => Fetch three best matches from Firestore RAG-powered Q&A with Genkit
53. }, inputSchema: inputSchema, outputSchema: z.string(), authPolicy: firebaseAuth((user) = > { }), }, async (input) => { const docs = await retrieve({ retriever: firestoreRetriever, query: input.question, options: {limit: 3}, }); const context = docs.map((doc) => doc.text()).join('\n\n'); const history = input.history ? . map((message) => `${message.role}: ${message.message}`).join('\n'); const prompt = ` Your name is Sofia, and you are a knowledge assistant. You were created by Peter Friese, who is a Developer Advocate on the Firebase team at Google. Peter created you to showcase how to use Firebase and AI to build a second brain. Purpose: * Utilize my personal knowledge base to answer any questions I have. Build context using the three best matching documents RAG-powered Q&A with Genkit
54. }, inputSchema: inputSchema, outputSchema: z.string(), authPolicy: firebaseAuth((user) = > { }), }, async (input) => { const docs = await retrieve({ retriever: firestoreRetriever, query: input.question, options: {limit: 3}, }); const context = docs.map((doc) => doc.text()).join('\n\n'); const history = input.history ? . map((message) => `${message.role}: ${message.message}`).join('\n'); const prompt = ` Your name is Sofia, and you are a knowledge assistant. You were created by Peter Friese, who is a Developer Advocate on the Firebase team at Google. Peter created you to showcase how to use Firebase and AI to build a second brain. Purpose: * Utilize my personal knowledge base to answer any questions I have. Build chat history RAG-powered Q&A with Genkit
55. query: input.question, options: {limit: 3}, }); const context = docs.map((doc) => doc.text()).join('\n\n'); const history = input.history ? . map((message) => `${message.role}: ${message.message}`).join('\n'); const prompt = ` Your name is Sofia, and you are a knowledge assistant. You were created by Peter Friese, who is a Developer Advocate on the Firebase team at Google. Peter created you to showcase how to use Firebase and AI to build a second brain. Purpose: * Utilize my personal knowledge base to answer any questions I have. Question: ${input.question} Behavior: 1. Analyze the context provided by me. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. The Q&A prompt RAG-powered Q&A with Genkit
56. query: input.question, options: {limit: 3}, }); const context = docs.map((doc) => doc.text()).join('\n\n'); const history = input.history ? . map((message) => `${message.role}: ${message.message}`).join('\n'); const prompt = ` Your name is Sofia, and you are a knowledge assistant. You were created by Peter Friese, who is a Developer Advocate on the Firebase team at Google. Peter created you to showcase how to use Firebase and AI to build a second brain. Purpose: * Utilize my personal knowledge base to answer any questions I have. Question: ${input.question} Behavior: 1. Analyze the context provided by me. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. The user’s question RAG-powered Q&A with Genkit
57. a second brain. Purpose: * Utilize my personal knowledge base to answer any questions I have. Question: ${input.question} Behavior: 1. Analyze the context provided by me. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. 4. If there is insufficient information or the knowledge base does not contain relevant information, clearly state you don't know the answer. 5. Do not generate responses based on assumptions or speculation. 6. Respond in a professional and helpful manner. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. Specifying model behavior RAG-powered Q&A with Genkit
58. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. 4. If there is insufficient information or the knowledge base does not contain relevant information, clearly state you don't know the answer. 5. Do not generate responses based on assumptions or speculation. 6. Respond in a professional and helpful manner. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. * Clear indication when an answer is not known. * No fabrication or speculation in responses. * Provide relevant code snippets. Context: ${context} History: ${history} ` More instructions RAG-powered Q&A with Genkit
59. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. 4. If there is insufficient information or the knowledge base does not contain relevant information, clearly state you don't know the answer. 5. Do not generate responses based on assumptions or speculation. 6. Respond in a professional and helpful manner. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. * Clear indication when an answer is not known. * No fabrication or speculation in responses. * Provide relevant code snippets. Context: ${context} History: ${history} ` Inject context (from the three documents) RAG-powered Q&A with Genkit
60. 2. Reference my personal knowledge base and gather relevant information. 3. Synthesize the information and provide a concise, accurate response. 4. If there is insufficient information or the knowledge base does not contain relevant information, clearly state you don't know the answer. 5. Do not generate responses based on assumptions or speculation. 6. Respond in a professional and helpful manner. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. * Clear indication when an answer is not known. * No fabrication or speculation in responses. * Provide relevant code snippets. Context: ${context} History: ${history} ` Inject history (from the chat) RAG-powered Q&A with Genkit
61. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. * Clear indication when an answer is not known. * No fabrication or speculation in responses. * Provide relevant code snippets. Context: ${context} History: ${history} ` const answer = await generate({model: gemini15Flash, prompt}); return answer.text(); } ); Send prompt to model RAG-powered Q&A with Genkit
62. 7. When I ask details about you as a person, just provide basic info, and then try to get the conversation back on track to talk about knowledge in my knowledgebase. 8. If I ask you about what you know, tell me that you only know what is in my knowledge base. Expectations: * Accurate and relevant responses. * Clear indication when an answer is not known. * No fabrication or speculation in responses. * Provide relevant code snippets. Context: ${context} History: ${history} ` const answer = await generate({model: gemini15Flash, prompt}); return answer.text(); } ); Return answer RAG-powered Q&A with Genkit
63. RAG-powered Q&A with Genkit private var functions = FirebaseConnector.shared.functions public func performQuery(question: String, history: [ChatMessage]? = nil) async -> Response { let semanticQAFlowCallable: Callable<Query, Response> = functions.httpsCallable("semanticQAFlow") do { let query = Query(question: question, history: history) let response = try await semanticQAFlowCallable(query) return response } catch { return Response(shortSummary: "I am sorry, but something went wrong.", answer: "Something went wrong. Please try again.", source: nil) } } Callable Cloud Function
64. RAG-powered Q&A with Genkit private var functions = FirebaseConnector.shared.functions public func performQuery(question: String, history: [ChatMessage]? = nil) async -> Response { let semanticQAFlowCallable: Callable<Query, Response> = functions.httpsCallable("semanticQAFlow") do { let query = Query(question: question, history: history) let response = try await semanticQAFlowCallable(query) return response } catch { return Response(shortSummary: "I am sorry, but something went wrong.", answer: "Something went wrong. Please try again.", source: nil) } } Invoke callable function
65. Demo
66. but…
67. “mhh hmm?”
68. @MainActor public struct SemanticQAIntent: @preconcurrency AppIntent { private var semanticQAService = SemanticQAService.shared @Parameter(title: "Question", description: "Answer from your knowledge base") var question: String? public static let title: LocalizedStringResource = "Search" static let description: LocalizedStringResource = "Search your saved artifacts" public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { Conform to AppIntent Siri: App Intents
69. @MainActor public struct SemanticQAIntent: @preconcurrency AppIntent { private var semanticQAService = SemanticQAService.shared @Parameter(title: "Question", description: "Answer from your knowledge base") var question: String? public static let title: LocalizedStringResource = "Search" static let description: LocalizedStringResource = "Search your saved artifacts" public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { Define input parameters Siri: App Intents
70. public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { throw $question.needsValueError("Sure - what would you like to know?") } let answer = await semanticQAService.performQuery(question: providedPhrase) return .result(dialog: IntentDialog(stringLiteral: "Here is your answer")) { Markdown(answer) .padding() .markdownBlockStyle(\.codeBlock) { configuration in configuration.label .relativeLineSpacing(.em(0.25)) .markdownTextStyle { FontFamilyVariant(.monospaced) FontSize(.em(0.85)) } Siri: App Intents This will be run when your Intent is invoked
71. public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { throw $question.needsValueError("Sure - what would you like to know?") } let answer = await semanticQAService.performQuery(question: providedPhrase) return .result(dialog: IntentDialog(stringLiteral: "Here is your answer")) { Markdown(answer) .padding() .markdownBlockStyle(\.codeBlock) { configuration in configuration.label .relativeLineSpacing(.em(0.25)) .markdownTextStyle { FontFamilyVariant(.monospaced) FontSize(.em(0.85)) } Siri: App Intents Call the Q&A Genkit flow
72. public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { throw $question.needsValueError("Sure - what would you like to know?") } let answer = await semanticQAService.performQuery(question: providedPhrase) return .result(dialog: IntentDialog(stringLiteral: "Here is your answer")) { Markdown(answer) .padding() .markdownBlockStyle(\.codeBlock) { configuration in configuration.label .relativeLineSpacing(.em(0.25)) .markdownTextStyle { FontFamilyVariant(.monospaced) FontSize(.em(0.85)) } Siri: App Intents Result is a dialog
73. public static var parameterSummary: some ParameterSummary { Summary("You asked: \(\.$question)") } public init() { } public func perform() async throws -> some ProvidesDialog & ShowsSnippetView { guard let providedPhrase = question else { throw $question.needsValueError("Sure - what would you like to know?") } let answer = await semanticQAService.performQuery(question: providedPhrase) return .result(dialog: IntentDialog(stringLiteral: "Here is your answer")) { Markdown(answer) .padding() .markdownBlockStyle(\.codeBlock) { configuration in configuration.label .relativeLineSpacing(.em(0.25)) .markdownTextStyle { FontFamilyVariant(.monospaced) FontSize(.em(0.85)) } Siri: App Intents gonzalezreal/swift-markdown-ui
74. Peter Friese, @peterfriese How I built a second brain Using Firebase, Gemini, Genkit, and Siri
75. S1 E4 Building a Second Brain App - Data Access 􀉉 Friday, November 15th 􀐫 18 : 00 CET / 9 : 00 PST / 12 : 00 EST / 17 : 00 UTC Livestream
76. peterfriese.dev @peterfriese youtube.com/c/PeterFriese/ not-only-swift.peterfriese.dev/ @peterfriese@iosdev.space youtube.com/c/Firebase Thanks!
77. Q & A