MM16NL - Import Data, lessons learned

I want to share my experiences I made in several projects with importing data. It’s manly a talk about some curious situations from my latest big project, where I struggled and how I mastered these situations

Transcript

1. Import Data lessons learned http://www.alboardman.com/ Meet-Magento NL, 13.May 2016
2. Marcel Hauri Magento Certified Developer 
 PHP for ages, Magento since 2013 
 Work for Stämpfli AG, Switzerland @mhauri
3. Funded in 1799 as a Printing House
4. PIM
5. Context ‣ multisite for 11 countries (websites)
 ‣ each country has its own publication
 ‣ each publication has informations in 11 languages (en_US / en_GB)
 ‣ currently ~2500 Products
 ‣ more than 1100 attributes in PIM http://www.amcharts.com/visited_countries/
6. Simplify where possible https://dribbble.com/marchukover
7. You won’t import over 1100 custom attributes in Magento Headline Simplify where possible
8. HTML List Headline You won’t import over 1100 custom attributes in Magento Simplify where possible
9. Only ~180 custom attributes used in Magento. 83.64% You won’t import over 1100 custom attributes in Magento Simplify where possible
10. Example from another Project Terrible Import Workflow (Time to import ~ 2.5h, Shop Offline during import) Over 2000 attribute sets and more than 9000 attributes Simplify where possible
11. Label / Value / Unit Simplify where possible
12. reduced to 200 custom attributes and 1 attribute set in total. 97.7% Over 2000 attribute sets and more than 9000 attributes Simplify where possible
13. Downsides Upsides Not Filterable Less atributes needed No Layered Navigation Less complicated Faster Import & Indexing Simplify where possible
14. Bulletproof your Import Process … http://markusmagnusson.tv/
15. is_dirty: true is_dirty: false delete by is_dirty = true Bulletproof your Import Process …
16. http://versiris.deviantart.com/ Bulletproof your Import Process …
17. is_dirty: true is_dirty: false delete by is_dirty = true Bulletproof your Import Process …
18. https://memez24.wordpress.com/ Bulletproof your Import Process …
19. Everyone makes mistakes! Mistake Driven Development - David Manners http://de.slideshare.net/Dutchento/david-manners-mm15nl
20. Everyone makes mistakes!
21. Logging – What’s Your Import Doing Right Now? http://electronicitems.tumblr.com/
22. 2015-08-19T06:20:02+00:00 INFO (6): ########################## INIT IMPORT ########################## 2015-08-19T06:20:02+00:00 INFO (6): Website: no_name (memory: 20.56 MB / peak: 20.62 MB) 2015-08-19T06:20:02+00:00 INFO (6): Root Category: United States (memory: 20.56 MB / peak: 20.62 MB) 2015-08-19T06:20:02+00:00 INFO (6): Import Category: Products (memory: 25.27 MB / peak: 25.54 MB) 2015-08-19T06:20:02+00:00 INFO (6): Store ID: 1 (memory: 25.27 MB / peak: 25.54 MB) 2015-08-19T06:20:06+00:00 INFO (6): Marked all imported categories for website no_name as dirty! (memory: 36.82 MB / peak: 52.52 MB) 2015-08-19T06:20:06+00:00 INFO (6): Category import initialized! (memory: 25.73 MB / peak: 52.52 MB) 2015-08-19T06:20:06+00:00 INFO (6): Load catalog values! (memory: 27.76 MB / peak: 52.52 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load overview values! (memory: 84.85 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load overview labels! (memory: 91.54 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load download labels! (memory: 92.04 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Preparing categories to import! (memory: 92.06 MB / peak: 173.60 MB) 2015-08-19T06:20:12+00:00 INFO (6): Cleanup memory! (memory: 37.23 MB / peak: 173.60 MB) 2015-08-19T06:20:20+00:00 INFO (6): Stored categories for level [1]. (memory: 44.18 MB / peak: 173.60 MB) 2015-08-19T06:20:21+00:00 INFO (6): Stored categories for level [2]. (memory: 45.50 MB / peak: 173.60 MB) 2015-08-19T06:20:24+00:00 INFO (6): Stored categories for level [3]. (memory: 46.96 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): Stored categories for level [4]. (memory: 44.31 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): Application examples import initialized! (memory: 42.02 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Segment categories set! (memory: 42.02 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Categories loaded! (memory: 44.12 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Application Examples loaded! (memory: 44.18 MB / peak: 173.60 MB) 2015-08-19T06:20:34+00:00 INFO (6): - Catalog values loaded! (memory: 101.43 MB / peak: 190.71 MB) 2015-08-19T06:20:52+00:00 INFO (6): Marked all imported products as unassigned in website ace_de! (memory: 149.11 MB / peak: 230.23 MB) 2015-08-19T06:21:02+00:00 INFO (6): Product import initialized! (memory: 25.40 MB / peak: 25.94 MB) 2015-08-19T06:21:03+00:00 INFO (6): Already imported products: 0 (memory: 28.61 MB / peak: 28.80 MB) 2015-08-19T06:21:03+00:00 INFO (6): Total products to import: 1604 (memory: 28.61 MB / peak: 28.80 MB) 2015-08-19T06:21:03+00:00 INFO (6): Start preparing product data! (memory: 31.39 MB / peak: 31.68 MB) 2015-08-19T06:21:06+00:00 INFO (6): - Product values loaded! (memory: 122.21 MB / peak: 246.36 MB) 2015-08-19T06:21:21+00:00 INFO (6): - Related products loaded! (memory: 124.92 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Catalog values loaded! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Default language set! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Store language set! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:25+00:00 INFO (6): - Overview values loaded! (memory: 198.29 MB / peak: 252.20 MB) 2015-08-19T06:21:25+00:00 INFO (6): - Overview labels loaded! (memory: 198.81 MB / peak: 252.20 MB) 2015-08-19T06:21:28+00:00 INFO (6): - Existing products loaded! (memory: 203.35 MB / peak: 252.20 MB) 2015-08-19T06:21:51+00:00 INFO (6): Cleanup memory! (memory: 90.46 MB / peak: 259.97 MB) 2015-08-19T06:21:51+00:00 INFO (6): Prepared 1604 products, start import now! (memory: 90.46 MB / peak: 259.97 MB) 2015-08-19T06:23:45+00:00 INFO (6): Product import done! (memory: 110.22 MB / peak: 259.97 MB) 2015-08-19T06:23:45+00:00 INFO (6): Category-Product-Relations initialized! (memory: 109.41 MB / peak: 259.97 MB) 2015-08-19T06:23:47+00:00 INFO (6): Found 215 categories. (memory: 45.86 MB / peak: 259.97 MB) 2015-08-19T06:23:50+00:00 INFO (6): Found 1464 products. (memory: 82.08 MB / peak: 259.97 MB) 2015-08-19T06:25:54+00:00 INFO (6): Category-Product-Relations set in 128.61 seconds! (memory: 545.52 MB / peak: 546.96 MB) 2015-08-19T06:20:31+00:00 INFO (6): All categories stored in 24.82 seconds! (memory: 44.31 MB / peak: 173.60 MB) 2015-08-19T06:20:36+00:00 INFO (6): All application examples stored in 5.63 seconds! (memory: 107.17 MB / peak: 190.71 MB) 2015-08-19T06:23:45+00:00 INFO (6): Imported 1604 of 1604 products in 162.29 seconds! (memory: 110.22 MB / peak: 259.97 MB) Logging – What’s Your Import Doing Right Now?
23. Load catalog values! (mem: 13.07 MB / peak:13.50 MB) Load overview values! (mem: 366.67 MB / peak: 962.75 MB) Logging – What’s Your Import Doing Right Now?
24. Conclusion • Simplify where possible! • Bulletproof your import process! • Don’t trust data! • Log as much information as possible! • Load only data you really need! • Everyone makes mistakes!
25. That’s all folks — Questions? twitter.com/mhauri http://giphy.com/gifs/zhiliao2-3o85xLfqj5TBqDXzuo
26. http://typicalhope.tumblr.com/