Import Data – from PIM and other systems – lessons learned

Import Data – from PIM and other systems – lessons learned

I want to share my experiences I made in several projects with importing data. It’s manly a talk about some curious situations from my latest big project, where I struggled and how I mastered these situations. #mm15ch

961dfec60de2d5e3338cc85f65f43391?s=128

Marcel Hauri

October 23, 2015
Tweet

Transcript

  1. Import Data – from PIM and other systems lessons learned

    http://www.alboardman.com/
  2. Marcel Hauri Magento Certified Developer! 
 PHP for ages, Magento

    for 2 years! 
 Work for Stämpfli AG @mhauri
  3. https://dribbble.com/daiduong

  4. Context ‣ multisite for 11 countries (websites)
 ‣ each country

    has its own publication
 ‣ each publication has informations in 11 languages (en_US / en_GB)
 ‣ currently ~2500 Products
 ‣ more than 800 attributes in PIM http://www.amcharts.com/visited_countries/
  5. Simplify where possible https://dribbble.com/marchukover

  6. Simplify where possible You won’t import over 800 custom attributes

    in Magento Headline
  7. Simplify where possible HTML List Headline You won’t import over

    800 custom attributes in Magento
  8. Simplify where possible ~180 custom attributes added in total. 77.5%

    You won’t import over 800 custom attributes in Magento
  9. Simplify where possible Over 2000 attribute sets and more than

    9000 attributes
  10. Simplify where possible Label / Value / Unit

  11. Simplify where possible reduced to 200 custom attributes and 1

    attribute set in total. 97.7% Over 2000 attribute sets and more than 9000 attributes
  12. Bulletproof your Import Process … http://markusmagnusson.tv/

  13. Bulletproof your Import Process …

  14. Bulletproof your Import Process … http://versiris.deviantart.com/

  15. Bulletproof your Import Process …

  16. Bulletproof your Import Process … https://memez24.wordpress.com/

  17. Everyone makes mistakes! Mistake Driven Development - David Manners http://de.slideshare.net/Dutchento/david-manners-mm15nl

  18. Logging – What’s Your Import Doing Right Now? http://electronicitems.tumblr.com/

  19. 2015-08-19T06:20:02+00:00 INFO (6): ########################## INIT IMPORT ##########################! 2015-08-19T06:20:02+00:00 INFO (6):

    Website: no_name (memory: 20.56 MB / peak: 20.62 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Root Category: United States (memory: 20.56 MB / peak: 20.62 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Import Category: Products (memory: 25.27 MB / peak: 25.54 MB)! 2015-08-19T06:20:02+00:00 INFO (6): Store ID: 1 (memory: 25.27 MB / peak: 25.54 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Marked all imported categories for website no_name as dirty! (memory: 36.82 MB / peak: 52.52 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Category import initialized! (memory: 25.73 MB / peak: 52.52 MB)! 2015-08-19T06:20:06+00:00 INFO (6): Load catalog values! (memory: 27.76 MB / peak: 52.52 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load overview values! (memory: 84.85 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load overview labels! (memory: 91.54 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Load download labels! (memory: 92.04 MB / peak: 173.60 MB)! 2015-08-19T06:20:09+00:00 INFO (6): Preparing categories to import! (memory: 92.06 MB / peak: 173.60 MB)! 2015-08-19T06:20:12+00:00 INFO (6): Cleanup memory! (memory: 37.23 MB / peak: 173.60 MB)! 2015-08-19T06:20:20+00:00 INFO (6): Stored categories for level [1]. (memory: 44.18 MB / peak: 173.60 MB)! 2015-08-19T06:20:21+00:00 INFO (6): Stored categories for level [2]. (memory: 45.50 MB / peak: 173.60 MB)! 2015-08-19T06:20:24+00:00 INFO (6): Stored categories for level [3]. (memory: 46.96 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): Stored categories for level [4]. (memory: 44.31 MB / peak: 173.60 MB)! ! 2015-08-19T06:20:31+00:00 INFO (6): Application examples import initialized! (memory: 42.02 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Segment categories set! (memory: 42.02 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Categories loaded! (memory: 44.12 MB / peak: 173.60 MB)! 2015-08-19T06:20:31+00:00 INFO (6): - Application Examples loaded! (memory: 44.18 MB / peak: 173.60 MB)! 2015-08-19T06:20:34+00:00 INFO (6): - Catalog values loaded! (memory: 101.43 MB / peak: 190.71 MB)! ! 2015-08-19T06:20:52+00:00 INFO (6): Marked all imported products as unassigned in website ace_de! (memory: 149.11 MB / peak: 230.23 MB)! 2015-08-19T06:21:02+00:00 INFO (6): Product import initialized! (memory: 25.40 MB / peak: 25.94 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Already imported products: 0 (memory: 28.61 MB / peak: 28.80 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Total products to import: 1604 (memory: 28.61 MB / peak: 28.80 MB)! 2015-08-19T06:21:03+00:00 INFO (6): Start preparing product data! (memory: 31.39 MB / peak: 31.68 MB)! 2015-08-19T06:21:06+00:00 INFO (6): - Product values loaded! (memory: 122.21 MB / peak: 246.36 MB)! 2015-08-19T06:21:21+00:00 INFO (6): - Related products loaded! (memory: 124.92 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Catalog values loaded! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Default language set! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:24+00:00 INFO (6): - Store language set! (memory: 156.60 MB / peak: 246.36 MB)! 2015-08-19T06:21:25+00:00 INFO (6): - Overview values loaded! (memory: 198.29 MB / peak: 252.20 MB)! 2015-08-19T06:21:25+00:00 INFO (6): - Overview labels loaded! (memory: 198.81 MB / peak: 252.20 MB)! 2015-08-19T06:21:28+00:00 INFO (6): - Existing products loaded! (memory: 203.35 MB / peak: 252.20 MB)! 2015-08-19T06:21:51+00:00 INFO (6): Cleanup memory! (memory: 90.46 MB / peak: 259.97 MB)! 2015-08-19T06:21:51+00:00 INFO (6): Prepared 1604 products, start import now! (memory: 90.46 MB / peak: 259.97 MB)! ! 2015-08-19T06:23:45+00:00 INFO (6): Product import done! (memory: 110.22 MB / peak: 259.97 MB)! 2015-08-19T06:23:45+00:00 INFO (6): Category-Product-Relations initialized! (memory: 109.41 MB / peak: 259.97 MB)! 2015-08-19T06:23:47+00:00 INFO (6): Found 215 categories. (memory: 45.86 MB / peak: 259.97 MB)! 2015-08-19T06:23:50+00:00 INFO (6): Found 1464 products. (memory: 82.08 MB / peak: 259.97 MB)! Logging – What’s Your Import Doing Right Now? 2015-08-19T06:20:31+00:00 INFO (6): All categories stored in 24.82 seconds! (memory: 44.31 MB / peak: 173.60 MB)! 2015-08-19T06:20:36+00:00 INFO (6): All application examples stored in 5.63 seconds! (memory: 107.17 MB / peak: 190.71 MB)! 2015-08-19T06:23:45+00:00 INFO (6): Imported 1604 of 1604 products in 162.29 seconds! (memory: 110.22 MB / peak: 259.97 MB)
  20. Load catalog values! (mem: 13.07 MB / peak:13.50 MB)! Load

    overview values! (mem: 366.67 MB / peak: 962.75 MB) Logging – What’s Your Import Doing Right Now?
  21. Conclusion • Simplify where possible! • Bulletproof your import process!

    • Don’t trust data! • Log as much information as possible! • Load only data you really need! • Everyone makes mistakes!
  22. That’s all folks — Questions? twitter.com/mhauri http://giphy.com/gifs/zhiliao2-3o85xLfqj5TBqDXzuo

  23. We're Hiring

  24. http://typicalhope.tumblr.com/