Engineering For Scale (By: Yasir Rizwan Saqib) - DevFest 2021

Talk by Yasif Rizwan Saqib (https://www.linkedin.com/in/yasir-rizwan-saqib-3911a02/) at DevFest Lahore 2021 by GDG Lahore.

Transcript

1. Yasir Rizwan Saqib ENGINEERING FOR SCALE SVP/CTO Google Devfest 01
2. Lessons from delivering 1000s of projects and converting 1000s of SEs into SSEs & PSEs. Google Devfest 02
3. LET’S TALK BIG MILLIONS of users TRILLIANS of transactions PETABYTES of data Hundreds & Thousands of LOC
4. A Little History - ASSESSMENT & STRATEGY PEOPLE. Assessing capacity and capability of the people PROCESS. Assessing the maturity of the processes PRODUCT. Assessing the Architecture & Design of the product Google Devfest 04
5. Architectures have been dehumanized for far too long Let’s give them a face. • Microservices • Containerization • Orchestration • Cloud native architectures • Auto scaling • Platform services * Fails only sometimes!!! 05
6. A Typical Architecture Diagram – We have Everything Google Devfest 06
7. They all seem to be. . . Beautiful Promising Functionally fine No usability issues Sufficiently Configurable, Adaptable, Extendible etc. High Performing and Apparently scalable too … Google Devfest 07
8. Under the Hood… It’s not the big stuff, but the little things that make the BIG difference. 08
9. WHAT REALLY BREAKS AT SCALE? 1. CONSISTENCY 2. PERFORMANCE Google Devfest 09
10. Its because Performance is IN YOUR FACE while Consistency is SNEAKY and is discussed and mourned in private. Why the world cares so much about Performance? Google Devfest 010
11. Performance Vs Consistency Everything is built for performance, but what about consistency? What are the tools for performance? • Scale up • Scale out • NLBs • SLBs What are the tools for consistency? • ORMs handle this pretty well. • Databases handle this for us. • Docker Swarm • GKE • Service Fabric…. Google Devfest 010
12. WE ARE USING ORM, NO? Google Devfest 013
13. What is Data Consistency? A Simple Example: Either order and payment both exist, or both do not exist. Else it is an inconsistent state. Order: No Order: Yes Order: No Order: Yes Payment: No Payment: Yes Payment: Yes Payment: No Consistent Consistent Inconsistent Inconsistent Google Devfest 012
14. CAUSES… BACK TO BASICS… Database Transaction Management. - Defining Transaction scope - Setting right isolation level Good Old Lost Update Problem (Multi-user Scenario). - Optimistic Locking (Versioning, Timestamping, Locking) - Pessimistic Locking (Locking) Concurrency Control (Multi-threaded Scenario). Finding Critical Sections Protecting Critical Sections Google Devfest 014
15. Atomic All or Nothing Consistency Obey all DB Rules Isolation Concurrent transaction do not affect Durability Guaranteed to be in storage TRANSACTIONS A C I D Google Devfest 015
16. Exclusive Lock (X) Will ensure that a page or row will be reserved exclusively for the transaction (INSERT, UPDATE, DELETE). One lock on a page or row. Shared Lock (S) Reserve page or Row for read Operation only. Multiple locks on a page or row by different transactions. Update Lock (U) Similar to (X). Adds an additional lock (U on the row that is in (S). Once data is ready for update, this lock changes from (U) to (X). Intent Lock (I) Intention to acquire lock in hierarchy objects. At Table Level and increases performance as there is no need to do operation on row level. Escalates to (X) or (S) depending upon the transaction. LOCKS Google Devfest 016
17. Read Committed Wait for the transaction to complete SELECT, please wait as an UPDATE is changing the data ISOLATION LEVELS Read Uncommitted Don’ t wait. Read the Dirty Data i.e. data that is changed but not committed yet. Read Committed With Snapshot Don’t wait. Take the snapshot at the start of the transaction. Read the snapshot Data i.e. the data that may be changed by another transaction in progress Repeatable Read UPDATE, please wait as my SELECT is reading the data Serializable Read Read Committed + Repeatable Read All wait until transaction is done. Big Performance issue Google Devfest 017
18. DEADLOCKS Process A Process B Resource Y Resource X Google Devfest 019
19. Unwanted Waits Use Read Uncommitted in reporting Queries RECOMMENDATIONS Data Changing within same transaction Use Repeatable Read or Read Committed in Queries to get accurate data. Queries might be slow. Transaction is Hanged Check Commit and Rollback in your transactions. Check for any open transaction in the database. Keep Transactions small. Deadlocks Keep Transactions small. Always log Deadlocks and check deadlock Graph / XML to find the issue. Use Read Uncommitted in reporting queries if possible. Try using resources in the same order. Retry on deadlock exceptions. Google Devfest 018
20. ACROSS MICROSERVICES TRANSACTIONS MANAGEMENT 1. EVENTUAL CONSISTENCY 2. ORCHESTRATORS Google Devfest 020
21. Any Questions? THANK YOU! Google Devfest 021