Behind the Scenes: DevFest 2019 KL Built Entirely on Firebase

Pretty sure by now you have used the DevFest website to get your tickets! Learn how Firebase powers DevFest website behind the scene. Learn about problems, techniques and solutions that were used to keep the DevFest running. Learn about the CI/CD process for Web and Cloud Functions, and how various Firebase features are used to handle features of the website, such as invite request, sending invites, to purchasing and reporting

Transcript

1. GDG Kuala Lumpur
2. Behind the Scenes: DevFest 2019 KL Built Entirely on Firebase Lim Shang Yi @shanggyilim GDG Kuala Lumpur
3. Name @handle GDG Kuala Lumpur
4. Name @handle GDG Kuala Lumpur
5. GDG Kuala Lumpur
6. Agenda 1. Challenges faced organising events at scale 2. How Firebase helps solve these challenges 3. Road bumps getting to production 4. Unexpected outcomes and lessons learnt Kuala Lumpur
7. https://directpoll.com/r?XD bzPBd3ixYqg865QmV0l4n 7eDtY4FnJ6YCPoUoV Quick Poll
8. Agenda 1. Challenges faced organising events at scale 2. How Firebase helps solve these challenges 3. Road bumps getting to production 4. Unexpected outcomes and lessons learnt Kuala Lumpur
9. Challenges Faced Organizing Events at Scale
10. Kuala Lumpur
11. Kuala Lumpur
12. But.. big events may not fit 1. Tickets in Peatix can only be scanned using Peatix App 2. Ticket List cannot be cross referenced with QR Code 3. Cannot tell information about the attendee Kuala Lumpur
13. But.. big events may not fit 4. Cannot add more tickets after cut-off date. 5. Consolidating attendees from attendees, VIPs, last minute attendees & sponsors tickets is messy Kuala Lumpur
14. We want to do more than just scanning ticket Kuala Lumpur
15. Kuala Lumpur
16. How Firebase helps solve these challenges
17. Kuala Lumpur Attendee Requests for a ticket Pay for the ticket Ticket on the website
18. Kuala Lumpur Ticket scanned at check in Ready to party!
19. Kuala Lumpur Collect stamps from sponsors Redeem items at GDGKL booth Complete survey for lucky draw
21. Firebase technologies used Cloud Pub/Sub Kuala Lumpur
22. Challenge 1: Make the application process seamless!
24. “anyone@gmail.com": { "gender":"male", "devPlatform":["web"], "email":"anyone@gmail.com", "shirtSize":"m", "createdOn":{"_seconds":1570458098,...}, "signedTnC":true, "food":"anything", "approved":false, "company":"Comapny", "fullName":"Lim Shang Yi", "timestamp":"2019-10-07T14:21:38.852Z", "excitedAbout":"talks", "jobRole":["fsd"] }
25. Challenge 2: Once approved, pay to get a confirmed ticket
26. application.status: “false” application.status: “true” Firestore
27. application.status: “false” application.status: “true” Firestore Background Cloud Functions firestore.collection(“tickets”) .doc(“your@email.com”).set({ status: “”, paymentState: “”, })
28. Http Cloud Functions firestore.collection(“tickets”) .doc(“your@email.com”).update({ status: “valid”, paymentState: “completed”, }) On successful payment
29. Http Cloud Functions firestore.collection(“tickets”) .doc(“your@email.com”).update({ status: “valid”, paymentState: “completed”, }) On successful payment
30. Challenge 3: Generate unique ticket id
31. Generating a unique identifier (Ticket ID) Random 3 digit number + increment counter Kuala Lumpur Prone to collision!! Prevent ticket id guessing
32. Using Sharding in Firestore for Sequence Numbers Kuala Lumpur Shards 1 Collection 2 0 3 4 Documents count: 128 count: 120 count: 129 count: 126 count: 123 Fields
33. Reducing probability of collision with randomized document fetch Kuala Lumpur Shards 1 Collection 2 0 3 4 Documents count: 128 count: 120 count: 129 count: 126 count: 123 Fields const shard_id = Math.floor(Math.random() * num_shards).toString();
34. Reducing probability of collision by increasing without fetching Kuala Lumpur count: 128 count: 120 count: 129 count: 126 count: 123 Fields const aShard = shardRef.data(); const increasedCount = aShard.count++; shreadRef.update(increasedCount) await shard_ref.update("count", firebase.firestore.FieldValue.increment(1));
35. Road bumps getting into Production
36. A app before going to production generally has: Kuala Lumpur • Way to handle environments (staging, production) • A continuous deployment pipeline to push features/fixes easily
37. Challenge 4: Handling multiple environments
38. DevFest has 2 “apps” Kuala Lumpur Single Page App Firebase managed backend - Firestore - Storage - Cloud Functions
39. How to manage environments with Firebase managed backends Kuala Lumpur • No easy way to handle environmental variables firebase functions:config:set stripe.api_key="$STRIPE_API_KEY" stripe.webhook_secret="$STRIPE_WEBHOOK_SECRET" gmail.client_secret="$GMAIL_CLIENT_SECRET" gmail.refresh_token="$GMAIL_REFRESH_TOKEN"
40. firebase functions:config:set stripe.api_key="$STRIPE_API_KEY" stripe.webhook_secret="$STRIPE_WEBHOOK_SECRET" gmail.client_secret="$GMAIL_CLIENT_SECRET" gmail.refresh_token="$GMAIL_REFRESH_TOKEN" import * as functions from "firebase-functions"; const stripeConfig = functions.config().stripe; const apiKey = stripeConfig.api_key;
41. Challenge 5: Keeping schema, security rules consistent across environments
42. Firestore migration script before function deployments start "functions": { "predeploy": [ "npm --prefix \"$RESOURCE_DIR\" run lint", "npm --prefix \"$RESOURCE_DIR\" run build", "npm --prefix \"$RESOURCE_DIR\" run migrate" ], "source": "functions" }, firebase.json "migrate": "node lib/migration.js",
43. Firestore security rules deployment firebase deploy --only firestore:rules firebase.json
44. Challenge 6: Creating a continuous deployment pipeline
46. #!/bin/bash -eo pipefail cd functions && ../node_modules/.bin/firebase deploy --only functions --token "$FIREBASE_TOKEN" === Deploying to 'devfest19-prod'... i deploying functions Running command: npm --prefix "$RESOURCE_DIR" run lint > functions@ lint /home/circleci/repo/functions > tslint --project tsconfig.json Running command: npm --prefix "$RESOURCE_DIR" run build > functions@ build /home/circleci/repo/functions > tsc
47. Running command: npm --prefix "$RESOURCE_DIR" run migrate > functions@ migrate /home/circleci/repo/functions > node lib/migration.js start migration migration_201909300004 already migrated. skipping. migration_201908251555 already migrated. skipping. migration_201909292304 already migrated. skipping. migration_201909202051 already migrated. skipping. migration_20190825 already migrated. skipping. migration succesful
48. ✔ functions[processSignUp(us-central1)]: Successful update operation. ✔ functions[unapproveAplications(asia-northeast1)]: Successful update operation. ✔ functions[updateTicketOnSurveyCreated(asia-northeast1)]: Successful update operation. ✔ functions[onPaymentSuccess(asia-northeast1)]: Successful update operation. ✔ functions[generateLuckyDrawList(asia-northeast1)]: Successful update operation. ✔ functions[createTicketOnApproved(asia-northeast1)]: Successful update operation. ✔ functions[generateTicketList(asia-northeast1)]: Successful update operation. ✔ functions[sendApplicationReceivedEmail(asia-northeast1)]: Successful update operation. ✔ functions[generateUnapprovedApplicationCsv(asia-northeast1)]: Successful update operatio ✔ functions[generateCsv(asia-northeast1)]: Successful update operation. ✔ functions[expireUnpaidTickets(asia-northeast1)]: Successful update operation. ✔ functions[onGenerateCheckoutSession(asia-northeast1)]: Successful update operation. ✔ functions[approveApplications(asia-northeast1)]: Successful update operation. ✔ functions[generateValidTicketList(asia-northeast1)]: Successful create operation. ✔ Deploy complete! Project Console: https://console.firebase.google.com/project/devfest19-prod/overview
49. #!/bin/bash -eo pipefail ./node_modules/.bin/firebase deploy --only firestore:rules --token "$FIREBASE_TOKEN" === Deploying to 'devfest19-prod'... i deploying firestore i firestore: checking firestore.rules for compilation errors... ✔ firestore: rules file firestore.rules compiled successfully i firestore: latest version of firestore.rules already up to date, skipping upload... ✔ firestore: released rules firestore.rules to cloud.firestore ✔ Deploy complete! Project Console: https://console.firebase.google.com/project/devfest19-prod/overview
51. Unexpected outcomes and lessons learnt
57. So what happened?
58. There were too many reads from Firestore, compared to the number of writes (request for ticket)
59. So what was the culprit?
60. Investigation 1: Unnecessary reads to Firestore • The main javascript loads all the information such as speakers, schedules, site info asynchronously in one shot. • Even if the speaker/schedule page is not visited, it is still loaded. Kuala Lumpur
61. Bug Fix 1: Moving the document reads to when its needed • Only read the ticket information at the registration page. • Only read the collection of speakers if navigated to the speaker page. • Only read the collection of schedule if navigated to the schedule page. Kuala Lumpur
62. Situation improved but not by much
63. So what was the biggest culprit?
64. The method of fetching data!
65. SPOT THE PROBLEM: Kuala Lumpur this.db.collection(key).get() .then(querySnapshot => { return querySnapshot.docs.map(doc => { const data = doc.data(); return { id: doc.id, ...data }; }); });
66. The problem is not the code. But incorrect usage!
68. So what's the problem !??
69. Using get() • Always fetch data from the server. • Data fetched everytime user navigates to a different page. Even if the data doesn’t change. Kuala Lumpur
70. Using onSnapshot() • Fetches the data from the server. • Data fetched is cached. • Loading the page multiple times will not incur document reads from Firestore. • Cache busting is performed automatically by onSnapshot() to get updated content Kuala Lumpur
72. Inefficient way to handle role and permissions
73. SPOT THE PROBLEM: Kuala Lumpur match /tickets/{email} { allow read: if request.auth.token != null && (request.auth.token.email == email || get(/databases/$(database)/documents /admin/$(email)).role == 'admin' ); } Every attempt to read the ticket will now cost addition al read
74. Using custom claims to handle user roles! functions.auth.user().onCreate(async (user) => { const adminSnapshot = await firebase.firestore() .collection("admin") .doc(user.email) .get(); await firebase.auth() .setCustomUserClaims(user.uid, customClaims); });
75. Using custom claims to handle user roles! match /tickets/{email} { allow read: if request.auth.token != null && (request.auth.token.email == email || request.auth.token.role == 'admin' ); }
76. Take aways!
77. Take aways • Firebase is good at doing things fast and at scale, whether right or wrong. • Monitor your document reads/writes counts when doing a user journey. Kuala Lumpur
78. Take aways • Firebase good way to get features / products off the ground real fast with scalability in mind. • Understand what your features do and pull only necessary data you need. Kuala Lumpur
79. Thank you! Lim Shang Yi @shanggyilim GDG Kuala Lumpur