MM16NL - Import Data, lessons learned

I want to share my experiences I made in several projects with importing data. It’s manly a talk about some curious situations from my latest big project, where I struggled and how I mastered these situations

Transcript

1. Import Data
   lessons learned
   http://www.alboardman.com/
   Meet-Magento NL, 13.May 2016
   

   View Slide

2. Marcel Hauri
   Magento Certified Developer
   

   PHP for ages, Magento since 2013
   

   Work for Stämpfli AG, Switzerland
   @mhauri
   

   View Slide

3. Funded in 1799 as a Printing House
   

   View Slide

4. PIM
   

   View Slide

5. Context
   ‣ multisite for 11 countries (websites)

   ‣ each country has its own publication

   ‣ each publication has informations in 11 languages (en_US / en_GB)

   ‣ currently ~2500 Products

   ‣ more than 1100 attributes in PIM
   http://www.amcharts.com/visited_countries/
   

   View Slide

6. Simplify where possible
   https://dribbble.com/marchukover
   

   View Slide

7. You won’t import over 1100 custom attributes in Magento
   Headline
   Simplify where possible
   

   View Slide

8. HTML List
   Headline
   You won’t import over 1100 custom attributes in Magento
   Simplify where possible
   

   View Slide

9. Only ~180 custom attributes used in Magento.
   83.64%
   You won’t import over 1100 custom attributes in Magento
   Simplify where possible
   

   View Slide

10. Example from another Project
    Terrible Import Workflow
    (Time to import ~ 2.5h, Shop Offline during import)
    Over 2000 attribute sets and more than 9000 attributes
    Simplify where possible
    

    View Slide

11. Label / Value / Unit
    Simplify where possible
    

    View Slide

12. reduced to 200 custom attributes and 1 attribute set in total.
    97.7%
    Over 2000 attribute sets and more than 9000 attributes
    Simplify where possible
    

    View Slide

13. Downsides Upsides
    Not Filterable Less atributes needed
    No Layered Navigation Less complicated
    Faster Import & Indexing
    Simplify where possible
    

    View Slide

14. Bulletproof your Import Process …
    http://markusmagnusson.tv/
    

    View Slide

15. is_dirty: true
    is_dirty: false
    delete by is_dirty = true
    Bulletproof your Import Process …
    

    View Slide

16. http://versiris.deviantart.com/
    Bulletproof your Import Process …
    

    View Slide

17. is_dirty: true
    is_dirty: false
    delete by is_dirty = true
    Bulletproof your Import Process …
    

    View Slide

18. https://memez24.wordpress.com/
    Bulletproof your Import Process …
    

    View Slide

19. Everyone makes mistakes!
    Mistake Driven Development - David Manners
    http://de.slideshare.net/Dutchento/david-manners-mm15nl
    

    View Slide

20. Everyone makes mistakes!
    

    View Slide

21. Logging – What’s Your Import Doing Right Now?
    http://electronicitems.tumblr.com/
    

    View Slide

22. 2015-08-19T06:20:02+00:00 INFO (6): ########################## INIT IMPORT ##########################
    2015-08-19T06:20:02+00:00 INFO (6): Website: no_name (memory: 20.56 MB / peak: 20.62 MB)
    2015-08-19T06:20:02+00:00 INFO (6): Root Category: United States (memory: 20.56 MB / peak: 20.62 MB)
    2015-08-19T06:20:02+00:00 INFO (6): Import Category: Products (memory: 25.27 MB / peak: 25.54 MB)
    2015-08-19T06:20:02+00:00 INFO (6): Store ID: 1 (memory: 25.27 MB / peak: 25.54 MB)
    2015-08-19T06:20:06+00:00 INFO (6): Marked all imported categories for website no_name as dirty! (memory: 36.82 MB / peak: 52.52 MB)
    2015-08-19T06:20:06+00:00 INFO (6): Category import initialized! (memory: 25.73 MB / peak: 52.52 MB)
    2015-08-19T06:20:06+00:00 INFO (6): Load catalog values! (memory: 27.76 MB / peak: 52.52 MB)
    2015-08-19T06:20:09+00:00 INFO (6): Load overview values! (memory: 84.85 MB / peak: 173.60 MB)
    2015-08-19T06:20:09+00:00 INFO (6): Load overview labels! (memory: 91.54 MB / peak: 173.60 MB)
    2015-08-19T06:20:09+00:00 INFO (6): Load download labels! (memory: 92.04 MB / peak: 173.60 MB)
    2015-08-19T06:20:09+00:00 INFO (6): Preparing categories to import! (memory: 92.06 MB / peak: 173.60 MB)
    2015-08-19T06:20:12+00:00 INFO (6): Cleanup memory! (memory: 37.23 MB / peak: 173.60 MB)
    2015-08-19T06:20:20+00:00 INFO (6): Stored categories for level [1]. (memory: 44.18 MB / peak: 173.60 MB)
    2015-08-19T06:20:21+00:00 INFO (6): Stored categories for level [2]. (memory: 45.50 MB / peak: 173.60 MB)
    2015-08-19T06:20:24+00:00 INFO (6): Stored categories for level [3]. (memory: 46.96 MB / peak: 173.60 MB)
    2015-08-19T06:20:31+00:00 INFO (6): Stored categories for level [4]. (memory: 44.31 MB / peak: 173.60 MB)
    2015-08-19T06:20:31+00:00 INFO (6): Application examples import initialized! (memory: 42.02 MB / peak: 173.60 MB)
    2015-08-19T06:20:31+00:00 INFO (6): - Segment categories set! (memory: 42.02 MB / peak: 173.60 MB)
    2015-08-19T06:20:31+00:00 INFO (6): - Categories loaded! (memory: 44.12 MB / peak: 173.60 MB)
    2015-08-19T06:20:31+00:00 INFO (6): - Application Examples loaded! (memory: 44.18 MB / peak: 173.60 MB)
    2015-08-19T06:20:34+00:00 INFO (6): - Catalog values loaded! (memory: 101.43 MB / peak: 190.71 MB)
    2015-08-19T06:20:52+00:00 INFO (6): Marked all imported products as unassigned in website ace_de! (memory: 149.11 MB / peak: 230.23 MB)
    2015-08-19T06:21:02+00:00 INFO (6): Product import initialized! (memory: 25.40 MB / peak: 25.94 MB)
    2015-08-19T06:21:03+00:00 INFO (6): Already imported products: 0 (memory: 28.61 MB / peak: 28.80 MB)
    2015-08-19T06:21:03+00:00 INFO (6): Total products to import: 1604 (memory: 28.61 MB / peak: 28.80 MB)
    2015-08-19T06:21:03+00:00 INFO (6): Start preparing product data! (memory: 31.39 MB / peak: 31.68 MB)
    2015-08-19T06:21:06+00:00 INFO (6): - Product values loaded! (memory: 122.21 MB / peak: 246.36 MB)
    2015-08-19T06:21:21+00:00 INFO (6): - Related products loaded! (memory: 124.92 MB / peak: 246.36 MB)
    2015-08-19T06:21:24+00:00 INFO (6): - Catalog values loaded! (memory: 156.60 MB / peak: 246.36 MB)
    2015-08-19T06:21:24+00:00 INFO (6): - Default language set! (memory: 156.60 MB / peak: 246.36 MB)
    2015-08-19T06:21:24+00:00 INFO (6): - Store language set! (memory: 156.60 MB / peak: 246.36 MB)
    2015-08-19T06:21:25+00:00 INFO (6): - Overview values loaded! (memory: 198.29 MB / peak: 252.20 MB)
    2015-08-19T06:21:25+00:00 INFO (6): - Overview labels loaded! (memory: 198.81 MB / peak: 252.20 MB)
    2015-08-19T06:21:28+00:00 INFO (6): - Existing products loaded! (memory: 203.35 MB / peak: 252.20 MB)
    2015-08-19T06:21:51+00:00 INFO (6): Cleanup memory! (memory: 90.46 MB / peak: 259.97 MB)
    2015-08-19T06:21:51+00:00 INFO (6): Prepared 1604 products, start import now! (memory: 90.46 MB / peak: 259.97 MB)
    2015-08-19T06:23:45+00:00 INFO (6): Product import done! (memory: 110.22 MB / peak: 259.97 MB)
    2015-08-19T06:23:45+00:00 INFO (6): Category-Product-Relations initialized! (memory: 109.41 MB / peak: 259.97 MB)
    2015-08-19T06:23:47+00:00 INFO (6): Found 215 categories. (memory: 45.86 MB / peak: 259.97 MB)
    2015-08-19T06:23:50+00:00 INFO (6): Found 1464 products. (memory: 82.08 MB / peak: 259.97 MB)
    2015-08-19T06:25:54+00:00 INFO (6): Category-Product-Relations set in 128.61 seconds! (memory: 545.52 MB / peak: 546.96 MB)
    2015-08-19T06:20:31+00:00 INFO (6): All categories stored in 24.82 seconds! (memory: 44.31 MB / peak: 173.60 MB)
    2015-08-19T06:20:36+00:00 INFO (6): All application examples stored in 5.63 seconds! (memory: 107.17 MB / peak: 190.71 MB)
    2015-08-19T06:23:45+00:00 INFO (6): Imported 1604 of 1604 products in 162.29 seconds! (memory: 110.22 MB / peak: 259.97 MB)
    Logging – What’s Your Import Doing Right Now?
    

    View Slide

23. Load catalog values! (mem: 13.07 MB / peak:13.50 MB)
    Load overview values! (mem: 366.67 MB / peak: 962.75 MB)
    Logging – What’s Your Import Doing Right Now?
    

    View Slide

24. Conclusion
    • Simplify where possible!
    • Bulletproof your import process!
    • Don’t trust data!
    • Log as much information as possible!
    • Load only data you really need!
    • Everyone makes mistakes!
    

    View Slide

25. That’s all folks — Questions?
    twitter.com/mhauri
    http://giphy.com/gifs/zhiliao2-3o85xLfqj5TBqDXzuo
    

    View Slide

26. http://typicalhope.tumblr.com/
    

    View Slide