Import Data – from PIM and other systems – lessons learned

I want to share my experiences I made in several projects with importing data. It’s manly a talk about some curious situations from my latest big project, where I struggled and how I mastered these situations. #mm15ch

Transcript

1. Import Data – from PIM and other systems lessons learned http://www.alboardman.com/
2. Marcel Hauri Magento Certified Developer! 
 PHP for ages, Magento for 2 years! 
 Work for Stämpfli AG @mhauri
3. https://dribbble.com/daiduong
4. Context ‣ multisite for 11 countries (websites)
 ‣ each country has its own publication
 ‣ each publication has informations in 11 languages (en_US / en_GB)
 ‣ currently ~2500 Products
 ‣ more than 800 attributes in PIM http://www.amcharts.com/visited_countries/
5. Simplify where possible https://dribbble.com/marchukover
6. Simplify where possible You won’t import over 800 custom attributes in Magento Headline
7. Simplify where possible HTML List Headline You won’t import over 800 custom attributes in Magento
8. Simplify where possible ~180 custom attributes added in total. 77.5% You won’t import over 800 custom attributes in Magento
9. Simplify where possible Over 2000 attribute sets and more than 9000 attributes
10. Simplify where possible Label / Value / Unit
11. Simplify where possible reduced to 200 custom attributes and 1 attribute set in total. 97.7% Over 2000 attribute sets and more than 9000 attributes
12. Bulletproof your Import Process … http://markusmagnusson.tv/
13. Bulletproof your Import Process …
14. Bulletproof your Import Process … http://versiris.deviantart.com/
15. Bulletproof your Import Process …
16. Bulletproof your Import Process … https://memez24.wordpress.com/
17. Everyone makes mistakes! Mistake Driven Development - David Manners http://de.slideshare.net/Dutchento/david-manners-mm15nl
18. Logging – What’s Your Import Doing Right Now? http://electronicitems.tumblr.com/
19. 2015-08-19T06:20:02+00:00 INFO (6): ########################## INIT IMPORT ##########################! 2015-08-19T06:20:02+00:00 INFO (6): Website: no_name (memory: 20.56 MB / peak: 20.62 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Root Category: United States (memory: 20.56 MB / peak: 20.62 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Import Category: Products (memory: 25.27 MB / peak: 25.54 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Store ID: 1 (memory: 25.27 MB / peak: 25.54 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Marked all imported categories for website no_name as dirty! (memory: 36.82 MB / peak: 52.52 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Category import initialized! (memory: 25.73 MB / peak: 52.52 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Load catalog values! (memory: 27.76 MB / peak: 52.52 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load overview values! (memory: 84.85 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load overview labels! (memory: 91.54 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load download labels! (memory: 92.04 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Preparing categories to import! (memory: 92.06 MB / peak: 173.60 MB)! 2015-08-19T06:20:12+00:00 INFO (6): Cleanup memory! (memory: 37.23 MB / peak: 173.60 MB)! 2015-08-19T06:20:20+00:00 INFO (6): Stored categories for level [1]. (memory: 44.18 MB / peak: 173.60 MB)! 2015-08-19T06:20:21+00:00 INFO (6): Stored categories for level [2]. (memory: 45.50 MB / peak: 173.60 MB)! 2015-08-19T06:20:24+00:00 INFO (6): Stored categories for level [3]. (memory: 46.96 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): Stored categories for level [4]. (memory: 44.31 MB / peak: 173.60 MB)! ! 2015-08-19T06:20:31+00:00 INFO (6): Application examples import initialized! (memory: 42.02 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Segment categories set! (memory: 42.02 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Categories loaded! (memory: 44.12 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Application Examples loaded! (memory: 44.18 MB / peak: 173.60 MB)! 2015-08-19T06:20:34+00:00 INFO (6): - Catalog values loaded! (memory: 101.43 MB / peak: 190.71 MB)! ! 2015-08-19T06:20:52+00:00 INFO (6): Marked all imported products as unassigned in website ace_de! (memory: 149.11 MB / peak: 230.23 MB)! 2015-08-19T06:21:02+00:00 INFO (6): Product import initialized! (memory: 25.40 MB / peak: 25.94 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Already imported products: 0 (memory: 28.61 MB / peak: 28.80 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Total products to import: 1604 (memory: 28.61 MB / peak: 28.80 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Start preparing product data! (memory: 31.39 MB / peak: 31.68 MB)! 2015-08-19T06:21:06+00:00 INFO (6): - Product values loaded! (memory: 122.21 MB / peak: 246.36 MB)! 2015-08-19T06:21:21+00:00 INFO (6): - Related products loaded! (memory: 124.92 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Catalog values loaded! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Default language set! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Store language set! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:25+00:00 INFO (6): - Overview values loaded! (memory: 198.29 MB / peak: 252.20 MB)! 2015-08-19T06:21:25+00:00 INFO (6): - Overview labels loaded! (memory: 198.81 MB / peak: 252.20 MB)! 2015-08-19T06:21:28+00:00 INFO (6): - Existing products loaded! (memory: 203.35 MB / peak: 252.20 MB)! 2015-08-19T06:21:51+00:00 INFO (6): Cleanup memory! (memory: 90.46 MB / peak: 259.97 MB)! 2015-08-19T06:21:51+00:00 INFO (6): Prepared 1604 products, start import now! (memory: 90.46 MB / peak: 259.97 MB)! ! 2015-08-19T06:23:45+00:00 INFO (6): Product import done! (memory: 110.22 MB / peak: 259.97 MB)! 2015-08-19T06:23:45+00:00 INFO (6): Category-Product-Relations initialized! (memory: 109.41 MB / peak: 259.97 MB)! 2015-08-19T06:23:47+00:00 INFO (6): Found 215 categories. (memory: 45.86 MB / peak: 259.97 MB)! 2015-08-19T06:23:50+00:00 INFO (6): Found 1464 products. (memory: 82.08 MB / peak: 259.97 MB)! Logging – What’s Your Import Doing Right Now? 2015-08-19T06:20:31+00:00 INFO (6): All categories stored in 24.82 seconds! (memory: 44.31 MB / peak: 173.60 MB)! 2015-08-19T06:20:36+00:00 INFO (6): All application examples stored in 5.63 seconds! (memory: 107.17 MB / peak: 190.71 MB)! 2015-08-19T06:23:45+00:00 INFO (6): Imported 1604 of 1604 products in 162.29 seconds! (memory: 110.22 MB / peak: 259.97 MB)
20. Load catalog values! (mem: 13.07 MB / peak:13.50 MB)! Load overview values! (mem: 366.67 MB / peak: 962.75 MB) Logging – What’s Your Import Doing Right Now?
21. Conclusion • Simplify where possible! • Bulletproof your import process! • Don’t trust data! • Log as much information as possible! • Load only data you really need! • Everyone makes mistakes!
22. That’s all folks — Questions? twitter.com/mhauri http://giphy.com/gifs/zhiliao2-3o85xLfqj5TBqDXzuo
23. We're Hiring
24. http://typicalhope.tumblr.com/