Pro Yearly is on sale from $80 to $50! »

High Performance Core Data

High Performance Core Data

http://highperformancecoredata.com/

Learn how to analyze, debug, and squeeze every last bit of performance out of Core Data. The standard Core Data implementation is very powerful and flexible but lacks performance. In this advanced session we will cover various performance analysis tools, model optimization, and various high performance concurrency models. This is an advanced discussion that assumes you have used Core Data before.

2345435ebf5920b1d600b7ee7d2221a1?s=128

Matthew Morey

November 15, 2013
Tweet

Transcript

  1. High Performance Core Data HighPerformanceCoreData.com MatthewMorey.com | @xzolian

  2. Agenda

  3. Managed Object Context Managed Object Managed Object Persistent Store Coordinator

    Persistent Store (SQLite, XML, ...) Model Main Queue
  4. Memory Speed Less More Slow Fast

  5. Memory Speed Less More Slow Fast

  6. Memory Speed Less More Slow Fast

  7. None
  8. None
  9. Tools

  10. #define TICK NSDate *startTime = [NSDate date] #define TOCK NSLog(@"Elapsed

    Time: %f", -[startTime timeIntervalSinceNow])
  11. #define TICK NSDate *startTime = [NSDate date] #define TOCK NSLog(@"Elapsed

    Time: %f", -[startTime timeIntervalSinceNow]) ... TICK; [self methodYouWantToMeasure]; TOCK;
  12. -com.apple.CoreData.SQLDebug [1,2,3]

  13. -com.apple.CoreData.SQLDebug [1,2,3]

  14. -com.apple.CoreData.SQLDebug [1,2,3]

  15. -com.apple.CoreData.SQLDebug [1,2,3]

  16. -com.apple.CoreData.SQLDebug [1,2,3] -com.apple.CoreData.SyntaxColoredLogging 1 -com.apple.CoreData.MigrationDebug 1 -com.apple.CoreData.SQLiteDebugSynchronous [0,1,2] -com.apple.CoreData.SQLiteIntegrityCheck [1]

    -com.apple.CoreData.ConcurrencyDebug [1,2,3]
  17. $ sqlite3

  18. $ sqlite3 UFO.sqlite sqlite> select * from sqlite_master; table|ZUFOSIGHTING|ZUFOSIGHTING|3|CREATE TABLE

    ZUFOSIGHTING ( Z_PK...VARCHAR ) table|Z_PRIMARYKEY|Z_PRIMARYKEY|4|CREATE TABLE Z_PRIMARYKEY (Z_ENT...INTEGER) table|Z_METADATA|Z_METADATA|5|CREATE TABLE Z_METADATA (Z_VERSION...BLOB)
  19. sqlite> SELECT t0.ZSHAPE, COUNT( t0.ZSHAPE ) FROM ZUFOSIGHTING t0 GROUP

    BY t0.ZSHAPE; changed|1 changing|1546 chevron|760 cigar|1782 circle|5271 cone|265 ... teardrop|595 triangle|6082 unknown|4490
  20. SQLite Professional Pony Debugger

  21. None
  22. None
  23. Tools •Macros •Launch Arguments •SQL •Instruments •Measure, Measure, Measure

  24. Fetching

  25. None
  26. NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; [fetchRequest setFetchBatchSize:20];

  27. None
  28. None
  29. NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; NSArray *UFOSightingsArray = [self.managedObjectContext executeFetchRequest:fetchRequest

    error:&error]; NSMutableDictionary *uniqueShapesDictionary = [NSMutableDictionary dictionary]; for (UFOSighting *sighting in UFOSightingsArray) { ... // Count unique shape items ... } NSManagedObject
  30. NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; [fetchRequest setResultType:NSDictionaryResultType]; [fetchRequest setPropertiesToFetch:@"shape"]; ...

    // Count unique shape items ... NSDictionaryResultType
  31. NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; [fetchRequest setPredicate: [NSPredicate predicateWithFormat: @"shape

    == %@", @"sphere"]]; NSUInteger sphereCount = [self.managedObjectContext countForFetchRequest:fetchRequest error:&error]; ... // Repeat for each unique shape ... countForFetchRequest:error:
  32. NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; NSExpressionDescription *expressionDescription = [[NSExpressionDescription alloc]

    init]; [expressionDescription setName:@"count"]; [expressionDescription setExpression: [NSExpression expressionForFunction:@"count:" arguments: @[[NSExpression expressionForKeyPath:@"shape"]]]]; [fetchRequest setPropertiesToFetch:@[@"shape", expressionDescription]]; [fetchRequest setPropertiesToGroupBy:@[@"shape"]]; [fetchRequest setResultType:NSDictionaryResultType]; NSExpression
  33. NSExpression NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; NSExpressionDescription *expressionDescription = [[NSExpressionDescription

    alloc] init]; [expressionDescription setName:@"count"]; [expressionDescription setExpression: [NSExpression expressionForFunction:@"count:" arguments: @[[NSExpression expressionForKeyPath:@"shape"]]]]; [fetchRequest setPropertiesToFetch:@[@"shape", expressionDescription]]; [fetchRequest setPropertiesToGroupBy:@[@"shape"]]; [fetchRequest setResultType:NSDictionaryResultType];
  34. None
  35. Prefetch Required Relationships

  36. [fetchRequest setRelationshipKeyPathsForPrefetching: @[@"photo"]];

  37. Fetch in the background

  38. [backgroundContext performBlock:^{ NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; fetchRequest.resultType = NSManagedObjectIDResultType;

    NSArray *managedObjectIDs = [backgroundContext executeFetchRequest:fetchRequest error:nil]; [mainQueueContext performBlock:^{ for (NSManagedObjectID *managedObjectID in managedObjectIDs) { UFOSighting *UFOSighting = [mainQueueContext objectWithID:managedObjectID]; // // Update UI on main queue // } }]; }];
  39. [backgroundContext performBlock:^{ NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; fetchRequest.resultType = NSManagedObjectIDResultType;

    NSArray *managedObjectIDs = [backgroundContext executeFetchRequest:fetchRequest error:nil]; [mainQueueContext performBlock:^{ for (NSManagedObjectID *managedObjectID in managedObjectIDs) { UFOSighting *UFOSighting = [mainQueueContext objectWithID:managedObjectID]; // // Update UI on main queue // } }]; }];
  40. [backgroundContext performBlock:^{ NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; fetchRequest.resultType = NSManagedObjectIDResultType;

    NSArray *managedObjectIDs = [backgroundContext executeFetchRequest:fetchRequest error:nil]; [mainQueueContext performBlock:^{ for (NSManagedObjectID *managedObjectID in managedObjectIDs) { UFOSighting *UFOSighting = [mainQueueContext objectWithID:managedObjectID]; // // Update UI on main queue // } }]; }];
  41. [backgroundContext performBlock:^{ NSFetchRequest *fetchRequest = [NSFetchRequest fetchRequestWithEntityName:@"UFOSighting"]; fetchRequest.resultType = NSManagedObjectIDResultType;

    NSArray *managedObjectIDs = [backgroundContext executeFetchRequest:fetchRequest error:nil]; [mainQueueContext performBlock:^{ for (NSManagedObjectID *managedObjectID in managedObjectIDs) { UFOSighting *UFOSighting = [mainQueueContext objectWithID:managedObjectID]; // // Update UI on main queue // } }]; }];
  42. NSFetchRequest *request = [NSFetchRequest fetchRequestWithEntityName:”MyEntity”]; NSAsynchronousFetchRequest *async = [[NSAsynchronousFetchRequest alloc]

    initWithFetchRequest:request completionBlock^(id result) { if (result.finalResult) { ... } }]; [context performBlock: ^() { NSError *error = nil; asyncResult = [moc executeRequest:asyncRequest error:&blockError]; }]; iOS 8 | OS X Yosemite
  43. Fetching •Don’t fetch more than you need •Set a batch

    size •Let SQLite do the calculations •Prefetch required relationships •Fetch in the background
  44. Predicates

  45. Light Comparisons First

  46. [fetchRequest setPredicate: [NSPredicate predicateWithFormat: @"shape == %@ AND duration >

    %i", @"sphere", 30]];
  47. [fetchRequest setPredicate: [NSPredicate predicateWithFormat: @"shape == %@ AND duration >

    %i", @"sphere", 30]]; [fetchRequest setPredicate: [NSPredicate predicateWithFormat: @"duration > %i AND shape == %@", 30, @"sphere"]];
  48. Be cautious with strings

  49. Predicate Costs Less More Beginswith Endswith Equality == Contains Matches

  50. Predicate Costs Less More Beginswith Endswith Equality == Contains Matches

    [cd] cost you even more
  51. Predicates •Do light (numerical) comparisons first •Beginswith/Endswith instead of Contains/Matches

    •Don’t use [cd]
  52. Searching

  53. Canonicalize String Properties

  54. Canonicalize String Properties Description canonicalizedDesc Green Mën green men BEAM

    beam Prόbë probe ABDUCTION abduction
  55. CFStringNormalize((CFMutableStringRef)result, kCFStringNormalizationFormD); CFStringFold((CFMutableStringRef)result, kCFCompareCaseInsensitive | kCFCompareDiacriticInsensitive | kCFCompareWidthInsensitive, NULL);

  56. Canonicalized Tokens

  57. None
  58. None
  59. componentsSeperatedByCharactersInSet

  60. Separate Stack

  61. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Primary Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML, ...) Model Search Tokens
  62. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Primary Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML, ...) Model Search Tokens URIRepresentation
  63. Hash of Tokens

  64. Searching •Canonicalize text properties •Use tokens •Separate persistence stack •Hash

    of tokens
  65. Data Model

  66. Don’t Overnormalize

  67. Use External Storage

  68. None
  69. None
  70. None
  71. None
  72. Data Model •Don’t overnormalize •Use external storage for large attributes

    •Large blobs as separate entity •Less data takes less time to fetch
  73. App Launch

  74. “Lots of Things” Problem

  75. NSBatchUpdateRequest *batchRequest = [NSBatchUpdateRequest batchUpdateRequestWithEntityName: [RSSItem entityName]]; batchRequest.propertiesToUpdate = @{NSStringFromSelector(@selector(read)):

    [NSNumber numberWithBool:YES]}; batchRequest.resultType = NSStatusOnlyResultType; batchRequest.predicate = [NSPredicate predicateWithFormat:@"..."]; NSError *requestError; NSBatchUpdateResult *result = (NSBatchUpdateResult *)[self.managedObjectContext executeRequest:batchRequest error:&requestError]; if (result == nil) { NSLog(@"Error: %@", [requestError localizedDescription]); } else { // Batch update succeeded } iOS 8 | OS X Yosemite
  76. Importing Data

  77. None
  78. Main Queue vs Private Queue self.managedObjectContext = self.persistenceStack.managedObjectContext; [self.managedObjectContext performBlockAndWait:^{

    [self import]; }];
  79. Main Queue vs Private Queue NSManagedObjectContext *privateManagedObjectContext = [[NSManagedObjectContext alloc]

    initWithConcurrencyType:NSPrivateQueueConcurrencyType]; [privateManagedObjectContext setPersistentStoreCoordinator: self.managedObjectContext.persistentStoreCoordinator]; [privateManagedObjectContext performBlockAndWait:^{ [self import]; }];
  80. None
  81. Typical Import Algorithm 1) JSON to NSDictionary 2) Find existing

    NSManagedObject 3) Optionally, create new NSManagedObject 4) Set attributes of new/updated NSManagedObject 5) Repeat for every JSON item
  82. Typical Import Algorithm 1) JSON to NSDictionary 2) Find existing

    NSManagedObject 3) Optionally, create new NSManagedObject 4) Set attributes of new/updated NSManagedObject 5) Repeat for every JSON item
  83. Efficient Import Algorithm 1) Sort import objects by ID 2)

    Execute a single fetch request for all matching IDs 3) Iterate through both import and existing objects 4) Update or create
  84. 1 4 2 3 ... New Data

  85. 1 4 2 3 ... New Data 4 1 ...

    Existing Data
  86. 1 4 2 3 ... New Data 4 1 ...

    Existing Data
  87. 1 4 2 3 ... New Data 4 1 ...

    Existing Data
  88. 1 4 2 3 ... New Data 4 1 ...

    Existing Data
  89. 1 2 3 4 ... New Data 1 4 ...

    Existing Data Update
  90. 1 2 3 4 ... New Data 1 4 ...

    Existing Data Update
  91. 1 2 3 4 ... New Data 1 4 ...

    Existing Data Insert
  92. 1 2 3 4 ... New Data 1 4 ...

    Existing Data Insert
  93. 2 1 2 3 4 ... New Data 1 4

    ... Existing Data
  94. 2 1 2 3 4 ... New Data 1 4

    ... Existing Data
  95. 1 2 3 4 ... New Data 1 4 ...

    Existing Data 2
  96. 1 2 3 4 ... New Data 1 4 ...

    Existing Data 2 Insert
  97. 1 2 3 4 ... New Data 1 4 ...

    Existing Data 2 Insert
  98. 3 1 2 3 4 ... New Data 1 4

    ... Existing Data 2
  99. 3 1 2 3 4 ... New Data 1 4

    ... Existing Data 2
  100. 1 2 3 4 ... New Data 1 4 ...

    Existing Data 2 3 Update
  101. 1 2 3 4 ... New Data 1 4 ...

    Existing Data 2 3 Update
  102. None
  103. Efficient Import Algorithm With Batching 1) Sort import objects by

    ID 2) Execute multiple fetch request for matching IDs - Iterate through both import and existing objects - Update or create 3) Save Batch
  104. // Grab sorted persisted managed objects, based on the batch

    NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] initWithEntityName:@"UFOSighting"]; NSPredicate *fetchPredicate = [NSPredicate predicateWithFormat: @"%K IN %@", @"GUID", jsonBatchGUIDArray];
  105. // Grab sorted persisted managed objects, based on the batch

    NSFetchRequest *fetchRequest = [[NSFetchRequest alloc] initWithEntityName:@"UFOSighting"]; NSPredicate *fetchPredicate = [NSPredicate predicateWithFormat: @"%K IN %@", @"GUID", jsonBatchGUIDArray];
  106. Pre-Populated SQLite File in App Bundle

  107. Download Pre-Populated SQLite File

  108. Importing Data •Do import work on private queues •Use efficient

    find-or-create algorithm •Work in batches to keep memory low •Use pre-populated SQLite file
  109. Memory

  110. [context refreshObject:UFOSightingManagedObject mergeChanges:NO];

  111. [context reset];

  112. Concurrency Models

  113. Managed Object Context Managed Object Managed Object Persistent Store Coordinator

    Persistent Store (SQLite, XML, ...) Model Main Queue
  114. Managed Object Context Managed Object Managed Object Persistent Store Coordinator

    Persistent Store (SQLite, XML, ...) Model Main Queue
  115. Managed Object Context Managed Object Managed Object Persistent Store Coordinator

    Persistent Store (SQLite, XML, ...) Model Main Queue
  116. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue
  117. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue
  118. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue
  119. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Context Did Save Notification Merge Changes From Notification
  120. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Context Did Save Notification Merge Changes From Notification
  121. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Context Did Save Notification Refetch and Reload
  122. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue
  123. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work
  124. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work Persisting to Disk
  125. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work Persisting to Disk
  126. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work Persisting to Disk
  127. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work Persisting to Disk
  128. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Managed Object Context Private Queue UI Work Persisting to Disk
  129. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue
  130. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates
  131. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates UI Work
  132. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates UI Work Persisting to Disk
  133. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates UI Work Persisting to Disk
  134. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates UI Work Persisting to Disk
  135. Persistent Store Coordinator Persistent Store Model Main Queue Managed Object

    Context Private Queue Managed Object Context Managed Object Context Private Queue Background Updates UI Work Persisting to Disk __block NSManagedObjectContext *temporaryContext = self.workerObjectContext; __block NSManagedObjectContext *managedObjectContext = self.managedObjectContext; __block NSManagedObjectContext *writerObjectContext = self.writerManagedObjectContext; [temporaryContext performBlock:^{ NSError *error = nil; if (![temporaryContext save:&error]) { // TODO: Handle error } [managedObjectContext performBlock:^{ NSError *error = nil; if (![managedObjectContext save:&error]) { // TODO: Handle error } [writerObjectContext performBlock:^{ NSError *error = nil; if (![writerObjectContext save:&error]) { // TODO: Handle error } // Success!!! }]; // writerObjectContext }]; // managedObjectContext }]; // temporaryContext
  136. None
  137. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model
  138. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model
  139. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model
  140. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model
  141. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model Context Did Save Notification
  142. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model Context Did Save Notification Merge Changes From Notification
  143. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model Context Did Save Notification Merge Changes From Notification
  144. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model Context Did Save Notification Refetch and Reload
  145. None
  146. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Persistent Store Coordinator Model Context Did Save Notification Refetch and Reload •Cannot pass objects between persistent store coordinators •Cannot pass objectIDs between persistent store coordinators •You can use URIRepresentation •You can use mergeChangesFromContextDidSaveNotification
  147. TLDR

  148. Managed Object Context Persistent Store Coordinator Persistent Store (SQLite, XML,

    ...) Model Managed Object Context Main Queue Private Queue Context Did Save Notification Refetch and Reload
  149. Concurrency Models •Use the simplest model that meets your needs

    •Update main context after large imports have completed
  150. Bang Head Here

  151. In general... •Use the tools, and measure, measure, measure •Don’t

    load more than you need to •Don’t use [cd] •Be smart with your data model •Import on a private queue in batches •Pick the correct (but simplest) concurrency model
  152. Questions? HighPerformanceCoreData.com MatthewMorey.com | @xzolian BuoyExplorer.com - Marine Conditions App

    WristPresenter.com - Presentation App ChaiOne.com - Hiring remote iOS engineers