ExtJS Buffered Store Internals

A presentation I gave at the Sencha Days in 2017. It explains the problem of using BufferedStores and dynamically adding and removing data, for which I found a solution that is available since then with https://github.com/coon-js/extjs-lib-core.

Transcript

1. ExtJS 6 Buffered Store Internals ExtJS 6
2. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album)
3. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album)
4. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album)
5. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album)
6. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Browsing = Information
7. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album)
8. ExtJS 6 1 .. n
9. ExtJS 6 1 .. n Pros: - easy to maintain - view on snapshots of data do not ask for much code wise - user stays focused Cons: - browsing through/operations on data require more user interaction - ux not consistent (e.g. data does not shift)
10. ExtJS 6 GridView (1902) Store RowSelectionModel Toolbar GridPanel CheckboxSelectionModel EditorGridPanel JsonReader DragZone // skip recalculating the row index if we are currently buffering, but not if we // are just pre-buffering // ... also, skip recalculating if the store is currently invalid
11. ExtJS 6 Challenges: Minor: - Rendering Performance (Solution: only render the rows which can be rendered into the currently visible rectangle) Major: Mirroring state of data was key! - Delete - Add - Update - MultiSelect (Range)
12. ExtJS 6 ExtJS 4: - BufferedRenderer (rendering performance) - BufferedStore
13. ExtJS 6 ExtJS 4: - BufferedRenderer (rendering performance) - BufferedStore add: function () { Ext.raise('add method may not be called on a buffered store - the store is a map of remote data'); }, insert: function () { Ext.raise('insert method may not be called on a buffered store - the store is a map of remote data'); },
14. ExtJS 6 add remove update
15. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Requirements • Grid should immediately show new data while keeping BufferedStore functionality • Grid should immediately show changes made to data • Grid should immediately update its View properly when data was removed Requirements • Grid should immediately show new data while keeping BufferedStore functionality • Grid should immediately show changes made to data • Grid should immediately update its View properly when data was removed
16. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Update • Does not change data set (i.e. number of data) • Changes data values, reflected by implemented renderer Update • Does not change data set (i.e. number of data) • Changes data values, reflected by implemented renderer Improvements - consider manual sort of data once data was changed (remoteSort : true), since this does not happen automatically
17. ExtJS 6 add remove update
18. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Add • Changes data set (i.e. number of data) • Challenge: Find data position in the local store and hope assume it is the same on the server (remote operations such as „sorting“ active) • Find data which needs to be swapped out by the newly added data (BufferedStore contains extracts from remote data) Add • Changes data set (i.e. number of data) • Challenge: Find data position in the local store and hope assume it is the same on the server (remote operations such as „sorting“ active) • Find data which needs to be swapped out by the newly added data (BufferedStore contains extracts from remote data)
19. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) AbstractStore ProxyStore Store JsonStore TreeStore ... BufferedStore
20. ExtJS 6 Store BufferedStore applyData : Ext.util.Collection add() insert() removeAt()
21. ExtJS 6 Store BufferedStore applyData : Ext.util.Collection applyData : Ext.data.PageMap add() insert() removeAt() Ext.util.HashMap Ext.util.LruCache
22. ExtJS 6 Ext.util.LruCache - maintains most recently accessed items and discards Least Recently Used (maxSize-property) - maps pages, not store data: scrolling → reusePage/loadPage/discardPage - adding/ removing data solely works on pages, not „store data“
23. ExtJS 6 Ext.util.LruCache store.loadPage(1); console.log(store.getData().map) 1: {prev: {…}, next: {…}, key: 1, value: Array(25)} 2: {prev: {…}, next: {…}, key: 2, value: Array(25)} 3: {prev: {…}, next: {…}, key: 3, value: Array(25)} 4: {prev: {…}, next: {…}, key: 4, value: Array(25)} 5: {prev: null, next: {…}, key: 5, value: Array(25)} ... 12: {prev: {…}, next: null, key: 12, value: Array(25)}
24. ExtJS 6 Ext.util.LruCache store.loadPage(1); console.log(store.getData().map) 1: {prev: {…}, next: {…}, key: 1, value: Array(25)} 2: {prev: {…}, next: {…}, key: 2, value: Array(25)} 3: {prev: {…}, next: {…}, key: 3, value: Array(25)} 4: {prev: {…}, next: {…}, key: 4, value: Array(25)} 5: {prev: null, next: {…}, key: 5, value: Array(25)} ... 12: {prev: {…}, next: null, key: 12, value: Array(25)} Pages Ext.data.Model
25. ExtJS 6 Ext.util.LruCache store.loadPage(1); console.log(store.getData().indexMap) {1 : 0, 2 : 1, 3 : 2, 4 : 3, 5 : 4, …, 300 : 299} <table id="tableview-1018-record-2" ... data-recordid="2" data-recordindex="3" class="...">
26. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Add (concrete) • Get Sorter – sort property, direction • Find insert index in LruCache.map: - iterate through pages - compare values based on sort information - if insert position found, add new record, remove one record (based on insert strategy, e.g. insert at page 2, index 5, remove record at position 25 to keep pageSize) • Update indexMap so view can work with newly added data • Refresh view Add (concrete) • Get Sorter – sort property, direction • Find insert index in LruCache.map: - iterate through pages - compare values based on sort information - if insert position found, add new record, remove one record (based on insert strategy, e.g. insert at page 2, index 5, remove record at position 25 to keep pageSize) • Update indexMap so view can work with newly added data • Refresh view
27. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; },
28. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]} map:
29. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]} 5:
30. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 5: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]}
31. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 5: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]}
32. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 5: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]}
33. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 5: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]}
34. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) addSorted : function(record) { var me = this, data = me.getData(), index = me.findInsertIndexInPagesForRecord(record), storeIndex = 0, page, pos, values; if (index === null) { return null; } page = index[0]; pos = index[1]; values = data.map[page].value; Ext.Array.splice(values, pos, 0, record); values.pop(); // Update the indexMap for (var startIdx in data.map) { for (var i = 0, len = data.map[startIdx].value.length; i < len; i++) { data.indexMap[data.map[startIdx].value[i].internalId] = storeIndex++; } } return record; }, map: 1: 2: 3: 5: 1 : {values : [...]}, 2 : {values : [...]}, 3 : {values : […]}, 5 : {values : […]} poof
35. ExtJS 6 add remove update
36. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) Requirements • Grid should immediately show new data while keeping BufferedStore functionality • Grid should immediately show changes made to data • Grid should immediately update its View properly when data was removed Requirements • Grid should immediately show new data while keeping BufferedStore functionality • Grid should immediately show changes made to data • Grid should immediately update its View properly when data was removed
37. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end
38. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end • Computing pages to reload • Does LRU discard or keep existing pages (add new, remove lru) • Backend Interaction • Asynchronous processes • Waiting for server response • User scrolls through data while request is active • Blocking input
39. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end • Computing pages to reload • Does LRU discard or keep existing pages • Backend Interaction • Asynchronous processes • Waiting for server response • User scrolls through data while request is active • Blocking input Fragmentation! LruCache-map may look like 1 2 3 11 12 21 22 23 24 ... Fragmentation! LruCache-map may look like 1 2 3 11 12 21 22 23 24 ...
40. ExtJS 6 „Struggle within, it suits you fine Struggle within, your ruin.“ - James Hetfield, Lars Ulrich The Struggle Within (The Black Album) LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end LruCache shifting problem • Pages 1-12 are available, page 1 is shown • Removing data in page 1 – solution: shift data down • Load data from server and append at the end • Computing pages to reload • Does LRU discard or keep existing pages • Backend Interaction • Asynchronous processes • Waiting for server response • User scrolls through data while request is active • Blocking input Fragmentation! LruCache-map may look like 1 2 3 11 12 21 22 23 24 ... Fragmentation! LruCache-map may look like 1 2 3 11 12 21 22 23 24 ... Improvements - after removing data, already buffered pages need to be purged once a new page loads 1 : 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 [VISIBLE] 2 : 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 DEL Page 1, id 3 SCROLL Page 3 (Request Parameters: Start 21, Limit 10) 1 : 1, 2, [3], 4, 5, 6, 7, 8, 9, 10 2 : 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 3 : 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 [VISIBLE] -> Record 21 missing since data shifted in the backend *[n] = marked as deleted, i.e. visible in the UI, but n/a from backend
41. ExtJS 6 add remove update