An Inside Look of IndexedDB Implementation in Gecko

Transcript

1. Inside IndexedDB Bevis Tseng
2. Outline • Recap: What’s IndexedDB? • Building Blocks in WebIDL • Building Blocks in Gecko • Database Operation Flows • Brief Introduction of Event Loop in HTML 5 • SQLite Operations/Schema for IndexedDB
3. What is IndexedDB? • Provides the ability to store large number of objects offline in UA with advanced indexing on specified object properties as keys for data retrieval. • A key has an associated type to one of: number, date, string, binary, or array. • http://w3c.github.io/IndexedDB/#introduction • https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm • https://docs.google.com/presentation/d/1U4KH1ZhBf4yZVN6wnam4KKF4SNxgVTlo2LckchRk6qE/edit?usp=sharing
4. Define Database Schema & Store data into DB var request = indexedDB.open("library"); request.onupgradeneeded = function() { // Define DB schema: var db = request.result; var store = db.createObjectStore("books", {keyPath: "isbn"}); var titleIndex = store.createIndex("by_title", "title", {unique: true}); var authorIndex = store.createIndex("by_author", "author"); // Populate with initial data. store.put({title: "Quarry Memories", author: "Fred", isbn: 123456}); store.put({title: "Water Buffaloes", author: "Fred", isbn: 234567}); store.put({title: "Bedrock Nights", author: "Barney", isbn: 345678}); };
5. Lookup Data by Selected Index and Cursor var txt = db.transaction("books", "readonly"); var store = txn.objectStore("books"); var index = store.index("by_author"); var request = index.openCursor(IDBKeyRange.only("Fred")); request.onsuccess = function() { var cursor = request.result; if (cursor) { || Called for each matching record. report(cursor.value.isbn, cursor.value.title, cursor.value.author); cursor.continue(); } else { || No more matching records. report(null); } };
6. Transactional Database db.createObjectStore("store", { autoIncrement: true }); trans1 = db.transaction(["store"], "readwrite"); store_t1 = trans1.objectStore("store"); store_t1.put("a"); // Will get key 1 store_t1.put("b"); // Will get key 2 trans1.abort(); // changes are rolled back after aborted // changes in trans2 will be started only if trans1 is finished. trans2 = db.transaction(["store"], "readwrite"); store_t2 = trans2.objectStore("store"); store_t2.put("c"); // Will get key 1 store_t2.put("d"); // Will get key 2
7. IDBTransaction Lifetime
8. INITIAL DONE No further DBRequests after script is complete LOADING txn.oncomplete() ::OnNewRequest():
 pendingRequestCount++ COMMITING ABORTING ::OnRequestFinished():
 pendingRequestCount-- ::OnNewRequest():
 pendingRequestCount++ ::OnRequestFinished() &&
 pendingRequestCount == 0 txn = db.transaction([“books”, “stores”], “readwrite”); request = txn.objectStore(“book”) .add({title: “C++ Primer”}) request.onsuccess()/onerror() txn.oncomplete() Transaction is aborted by the script or UA unexpectedly txn.onabort() txn.abort()
9. WebIDL Building Block
10. <interface> IDBFactory open(name, version): IDBOpenRequest deleteDatabase(name): IDBOpenRequest cmp(first, second): short <interface> IDBEnvironment indexedDB: IDBFactory <interface> Window <interface> WorkerGlobalScope <interface> IDBDatabase name: DOMString version: long long objectStoreNames: DOMStringList createObjectStore(name, option): IDBObjectStore deleteObjectStore(name): void transaction(storeNames, mode): IDBTransaction onabort onerror onversionchange <interface> IDBTransaction objectStoreNames: DOMStringList mode: IDBTransactionMode db: IDBDatabase error: DOMException objectStore(name): IDBObjectStore abort: void onabort oncomplete onerror <interface> IDBObjectStore name: DOMString keyPath: any indexedNames: DOMStringList transaction: IDBTransaction autoIncrement: boolean put|add|delete|clear|get|getAll| getAllKeys|count|openCursor| openKeyCursor: IDBRequest createIndex(name, keyPath, options): IDBIndex deleteIndex(name): void index(name): IDBIndex <interface> IDBKeyRange lower, upper: any lowerOpen, upperOpen: boolean only(value) lowerBound(lower, open): IDBKeyRange upperBound(upper, open): IDBKeyRange bound(lower, upper, lowerOpen, upperOpen): IDBKeyRange include(key): boolean <interface> IDBIndex name: DOMString objectStore: IDBObjectStore keyPath: any multiEntry, unique: boolean get|getKey(key): IDBRequest getAll|getAllKey(query, count): IDBRequest openCursor|openKeyCursor(query, direction): IDBRequest count(query): IDBRequest
11. <interface> IDBCursor source: IDBObjectStore or IDBIndex direction: IDBCursorDirection key: any primaryKey: any advance(count) continue(key) continuePrimaryKey(key, primaryKey) update(value): IDBRequest delete(): IDBRequest <interface> IDBCursorWithValue value: any <interface> IDBRequest result: any error: DOMException source: IDBObjectStore|IDBIndex| IDBCursor transaction: IDBTransaction readyState: IDBRequestReadyState onsuccess onerror <interface> IDBOpenRequest onblocked onupgradeneeded <interface> IDBVersionChangeEvent oldVersion newVersion
12. Event sequence when aborting transaction IDBRequest.onerror -> IDBTransaction.onerror -> IDBTransaction.onabort http://w3c.github.io/IndexedDB/#abort-transaction
13. Building Block inside Gecko
14. IDB WebIDL Implementation PBackground IPDL Protocol Factory Database Normal Transaction VersionChange Transaction Cursor Database File Connection Pool Database Connection Database Connection MozStorageService MozStorage Connection Permission Manager Quota Manager … Quota Client
15. PBackground utilized in IndexedDB http://blog.kylehuey.com/post/136679681701/what-is-pbackground https://developer.mozilla.org/en-US/docs/Mozilla/IPDL/PBackground https://drive.google.com/a/mozilla.com/file/d/0B9z1LrL-6YCYY2xMVFAzNDU5YVU/view
16. Worker Thread#2 Worker Thread#1 Main Thread PBackground Thread FactoryParent# FactoryChild#1 FactoryParent# FactoryChild#2 Worker Thread#1 FactoryParent# FactoryParent# FactoryChild#1 FactoryChild#2 Parent Process Main Thread Worker Thread#1 FactoryParent# FactoryParent# FactoryParent# FactoryParent# FactoryChild#5 FactoryChild#6 FactoryChild#7 FactoryChild#8 Main Thread Worker Thread#1 FactoryParent# FactoryParent# FactoryChild#9 FactoryChild#10 Child Process#1 Child Process#2
17. Type of Database Operations
18. VersionChange TransactionOp FactoryOp Open DatabaseOp Delete DatabaseOp Database OperationBase (Create|Delete|Rename) ObjectStoreOp (Create|Delete|Rename) IndexOp Normal TransactionOp IndexRequestOpBase ObjectStore AddOrPutRequestOp ObjectStore GetAllKeysRequestOp ObjectStore ClearRequestOp ObjectStore GetRequestOp ObjectStore DeleteRequestOp ObjectStore CountRequestOp Index GetRequestOp Index GetKeyRequestOp Index CountRequestOp Cursor:: CursorOp Cursor:: OpenOp Cursor:: ContinueOp IDBIndex.getAll() and IDBIndex.getAllKeys() are taken care by GetRequestOp and GetKeyRequestOp respectively IDBCursor.update() and IDBCursor.delete() utilize ObjectStoreAddOrPutRequestOp and ObjectStoreDeleteRequestOp respectively with primary key specified TransactionDatabase OperationBase
19. Generic Flow of Factory Operations
20. Initial Permission Challenge Send Results Completed Directory & Database Open Begin Version Change Database Work Same to requested version 1. Start Permission Challenge if persistent storage type is specified. 2. Go to next state if permission granted from user. 3. Otherwise, go to SendResults Requested from IDBFactory.open() or IDBFactory.delete() 1. Ask QuotaManager to Create/Open directory and database files as followed: • Directory in • $profile/storage/(default|permanent|temporary)/$origin/idb/ • …./$origin/idb/$SHUFFLED_NAME.files/ • SQLite File: • …./$origin/idb/$SHUFFLED_NAME.sqlite • … 1. Go to BeginVersionChange, if • Open with requested version > stored version • Or deletion operation is requested 2. Otherwise, go to SendResults Not granted 1. If other connections are alive, • send versionChange event to these connections and • trigger IDBOpenRequest.onblocked(). 2. Wait for other connections to close. 3. For Open request, • Create a VersionChangeTransaction for the upgrade process of DB schema. • Trigger IDBOpenRequest.onupgradeneeded(). 4. For Delete request, Clear all IDB files from $origin/idb/ folder. 1. This eventually triggers • IDBOpenRequest.onsuccess() with IDBDatabase instance. • IDBOpenRequest.onerror() with proper error cause. Note: Multiple threads are invoked during this operation in different states, including: PBackground Thread, IO Thread, Connection Thread, Main Thread Error
21. Generic Flow of Transaction Database Operations
22. Initial RunOnConnectionThread RunOnOwningThread DoDatabaseWork SendSuccessResult SendFailureResult 1. Operation is created after receiving IDBRequest from IDBObjectStore, IDBIndex or IDBCursor. (PBackground) 2.Dispatch itself into ConnectionPool (ConnectionThread) 3.Run SQL Statement via DatabaseConnection 4.Dispatch itself back to owning thread. (PBackground) 5.Send Operation Result back to IDBRequest
23. PBackground Implementation in ActorsParent
24. PBackground IDBFactoryParent Factory PBackground IDBDatabaseParent Database PBackground IDBTransactionParent NormalTransaction PBackground IDBVersionChangeTransactionParent VersionChangeTransaction PBackground IDBDatabaseFileParent DatabaseFile PBackground IDBCursorParent Cursor PBackground IDBUtilsParent Utils PBackground IDBDatabaseRequestParent DatabaseOp PBackground IDBFactoryRequestParent FactoryOp PBackground IDBRequestParent NormalTransactionOp
25. Connection Pool • A Pool of DatabaseConnections • Each owns a MozStorageConnection for the access of SQLite DB. • ConnectionPool::GetOrCreateConnection() • Each connection handle all the IDBRequests of an open database in a connection thread assigned to the database. • ConnectionPool::Start() • ConnectionPool:ScheuleTransaction() • Maximal Number of Connection Threads = 20.
26. Event Loop: NS_DispatchToCurrentThread() v.s. RunInStableState() v.s RunInMetastableState() • How to identify that there is no request after transaction is created at the next tick of the event loop? • https://jakearchibald.com/2015/tasks-microtasks-queues-and-schedules/ • https://www.w3.org/TR/html5/webappapis.html#processing-model-3
27. Stable State Metastable State Metastable State Metastable State Metastable State Metastable State Task#1 Task#2 Task#3 Event Queue Ex: DOM Event dispatched asynchronously MicroTask#1 MicroTask#2 MicroTask#3 MicroTask#4 MicroTask Queue Ex: Resolved Promise Note: 1. Metastable State is gecko-only. (Not specified in HTML 5. See Bug 1179909.) 2. See more discussion of microtasks and promises in Bug 1193394. 3. Stable State was removed from the steps of the latest #whatwg-eventloop-processing-model.
28. TRANSACTION in SQLite • Changes during the lifetime of an IDBTransaction are cancellable and isolated from other transactions.
29. SQLite TRANSACTION in IDB implementation • For each !ReadOnly Transaction, • BEGIN IMMEDIATE TRANSACTION • StartTransactionOp if !ReadOnly mode, VersionChangeOp. • DatabaseConnection::BeginWriteTransaction • COMMIT (CommitOp with success result) • DatabaseConnection::CommitWriteTransaction • ROLLBACK (CommitOp with failure result) • DatabaseConnection::RollbackWriteTransaction • Fore each write IDBRequest in a Transaction, • SAVE_POINT sp; • DatabaseConnection::StartSavePoint • RELEASE SAVE_POINT sp; • DatabaseConnection::ReleaseSavePoint • ROLLBACK TO SAVE_POINT sp; • DatabaseConnection::RollbackSavePoint
30. IDB Schema in SQLite DB Readable via SQLite Database Browser(http://sqlitebrowser.org/) to the .sqlite file in • Linux: $HOME/.mozilla/firefox/$profile/storage/default/$origin/idb/ • OS X: $HOME/Library/Application Support/Firefox/$profile/storage/default/$origin/idb/ Note: unique_index_data set (index_id, value) as PRIMARY KEY to ensure the value of the index key in each entity is unique.