Tweet
Tweet

Slide 1

Slide 1 text

Import Data lessons learned http://www.alboardman.com/ Meet-Magento NL, 13.May 2016

Slide 2

Slide 2 text

Marcel Hauri Magento Certified Developer 
 PHP for ages, Magento since 2013 
 Work for Stämpfli AG, Switzerland @mhauri

Slide 3

Slide 3 text

Funded in 1799 as a Printing House

Slide 4

Slide 4 text

PIM

Slide 5

Slide 5 text

Context ‣ multisite for 11 countries (websites)
 ‣ each country has its own publication
 ‣ each publication has informations in 11 languages (en_US / en_GB)
 ‣ currently ~2500 Products
 ‣ more than 1100 attributes in PIM http://www.amcharts.com/visited_countries/

Slide 6

Slide 6 text

Simplify where possible https://dribbble.com/marchukover

Slide 7

Slide 7 text

You won’t import over 1100 custom attributes in Magento Headline Simplify where possible

Slide 8

Slide 8 text

HTML List Headline You won’t import over 1100 custom attributes in Magento Simplify where possible

Slide 9

Slide 9 text

Only ~180 custom attributes used in Magento. 83.64% You won’t import over 1100 custom attributes in Magento Simplify where possible

Slide 10

Slide 10 text

Example from another Project Terrible Import Workflow (Time to import ~ 2.5h, Shop Offline during import) Over 2000 attribute sets and more than 9000 attributes Simplify where possible

Slide 11

Slide 11 text

Label / Value / Unit Simplify where possible

Slide 12

Slide 12 text

reduced to 200 custom attributes and 1 attribute set in total. 97.7% Over 2000 attribute sets and more than 9000 attributes Simplify where possible

Slide 13

Slide 13 text

Downsides Upsides Not Filterable Less atributes needed No Layered Navigation Less complicated Faster Import & Indexing Simplify where possible

Slide 14

Slide 14 text

Bulletproof your Import Process … http://markusmagnusson.tv/

Slide 15

Slide 15 text

is_dirty: true is_dirty: false delete by is_dirty = true Bulletproof your Import Process …

Slide 16

Slide 16 text

http://versiris.deviantart.com/ Bulletproof your Import Process …

Slide 17

Slide 17 text

is_dirty: true is_dirty: false delete by is_dirty = true Bulletproof your Import Process …

Slide 18

Slide 18 text

https://memez24.wordpress.com/ Bulletproof your Import Process …

Slide 19

Slide 19 text

Everyone makes mistakes! Mistake Driven Development - David Manners http://de.slideshare.net/Dutchento/david-manners-mm15nl

Slide 20

Slide 20 text

Everyone makes mistakes!

Slide 21

Slide 21 text

Logging – What’s Your Import Doing Right Now? http://electronicitems.tumblr.com/

Slide 22

Slide 22 text

2015-08-19T06:20:02+00:00 INFO (6): ########################## INIT IMPORT ########################## 2015-08-19T06:20:02+00:00 INFO (6): Website: no_name (memory: 20.56 MB / peak: 20.62 MB) 2015-08-19T06:20:02+00:00 INFO (6): Root Category: United States (memory: 20.56 MB / peak: 20.62 MB) 2015-08-19T06:20:02+00:00 INFO (6): Import Category: Products (memory: 25.27 MB / peak: 25.54 MB) 2015-08-19T06:20:02+00:00 INFO (6): Store ID: 1 (memory: 25.27 MB / peak: 25.54 MB) 2015-08-19T06:20:06+00:00 INFO (6): Marked all imported categories for website no_name as dirty! (memory: 36.82 MB / peak: 52.52 MB) 2015-08-19T06:20:06+00:00 INFO (6): Category import initialized! (memory: 25.73 MB / peak: 52.52 MB) 2015-08-19T06:20:06+00:00 INFO (6): Load catalog values! (memory: 27.76 MB / peak: 52.52 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load overview values! (memory: 84.85 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load overview labels! (memory: 91.54 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Load download labels! (memory: 92.04 MB / peak: 173.60 MB) 2015-08-19T06:20:09+00:00 INFO (6): Preparing categories to import! (memory: 92.06 MB / peak: 173.60 MB) 2015-08-19T06:20:12+00:00 INFO (6): Cleanup memory! (memory: 37.23 MB / peak: 173.60 MB) 2015-08-19T06:20:20+00:00 INFO (6): Stored categories for level [1]. (memory: 44.18 MB / peak: 173.60 MB) 2015-08-19T06:20:21+00:00 INFO (6): Stored categories for level [2]. (memory: 45.50 MB / peak: 173.60 MB) 2015-08-19T06:20:24+00:00 INFO (6): Stored categories for level [3]. (memory: 46.96 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): Stored categories for level [4]. (memory: 44.31 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): Application examples import initialized! (memory: 42.02 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Segment categories set! (memory: 42.02 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Categories loaded! (memory: 44.12 MB / peak: 173.60 MB) 2015-08-19T06:20:31+00:00 INFO (6): - Application Examples loaded! (memory: 44.18 MB / peak: 173.60 MB) 2015-08-19T06:20:34+00:00 INFO (6): - Catalog values loaded! (memory: 101.43 MB / peak: 190.71 MB) 2015-08-19T06:20:52+00:00 INFO (6): Marked all imported products as unassigned in website ace_de! (memory: 149.11 MB / peak: 230.23 MB) 2015-08-19T06:21:02+00:00 INFO (6): Product import initialized! (memory: 25.40 MB / peak: 25.94 MB) 2015-08-19T06:21:03+00:00 INFO (6): Already imported products: 0 (memory: 28.61 MB / peak: 28.80 MB) 2015-08-19T06:21:03+00:00 INFO (6): Total products to import: 1604 (memory: 28.61 MB / peak: 28.80 MB) 2015-08-19T06:21:03+00:00 INFO (6): Start preparing product data! (memory: 31.39 MB / peak: 31.68 MB) 2015-08-19T06:21:06+00:00 INFO (6): - Product values loaded! (memory: 122.21 MB / peak: 246.36 MB) 2015-08-19T06:21:21+00:00 INFO (6): - Related products loaded! (memory: 124.92 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Catalog values loaded! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Default language set! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:24+00:00 INFO (6): - Store language set! (memory: 156.60 MB / peak: 246.36 MB) 2015-08-19T06:21:25+00:00 INFO (6): - Overview values loaded! (memory: 198.29 MB / peak: 252.20 MB) 2015-08-19T06:21:25+00:00 INFO (6): - Overview labels loaded! (memory: 198.81 MB / peak: 252.20 MB) 2015-08-19T06:21:28+00:00 INFO (6): - Existing products loaded! (memory: 203.35 MB / peak: 252.20 MB) 2015-08-19T06:21:51+00:00 INFO (6): Cleanup memory! (memory: 90.46 MB / peak: 259.97 MB) 2015-08-19T06:21:51+00:00 INFO (6): Prepared 1604 products, start import now! (memory: 90.46 MB / peak: 259.97 MB) 2015-08-19T06:23:45+00:00 INFO (6): Product import done! (memory: 110.22 MB / peak: 259.97 MB) 2015-08-19T06:23:45+00:00 INFO (6): Category-Product-Relations initialized! (memory: 109.41 MB / peak: 259.97 MB) 2015-08-19T06:23:47+00:00 INFO (6): Found 215 categories. (memory: 45.86 MB / peak: 259.97 MB) 2015-08-19T06:23:50+00:00 INFO (6): Found 1464 products. (memory: 82.08 MB / peak: 259.97 MB) 2015-08-19T06:25:54+00:00 INFO (6): Category-Product-Relations set in 128.61 seconds! (memory: 545.52 MB / peak: 546.96 MB) 2015-08-19T06:20:31+00:00 INFO (6): All categories stored in 24.82 seconds! (memory: 44.31 MB / peak: 173.60 MB) 2015-08-19T06:20:36+00:00 INFO (6): All application examples stored in 5.63 seconds! (memory: 107.17 MB / peak: 190.71 MB) 2015-08-19T06:23:45+00:00 INFO (6): Imported 1604 of 1604 products in 162.29 seconds! (memory: 110.22 MB / peak: 259.97 MB) Logging – What’s Your Import Doing Right Now?

Slide 23

Slide 23 text

Load catalog values! (mem: 13.07 MB / peak:13.50 MB) Load overview values! (mem: 366.67 MB / peak: 962.75 MB) Logging – What’s Your Import Doing Right Now?

Slide 24

Slide 24 text

Conclusion • Simplify where possible! • Bulletproof your import process! • Don’t trust data! • Log as much information as possible! • Load only data you really need! • Everyone makes mistakes!

Slide 25

Slide 25 text

That’s all folks — Questions? twitter.com/mhauri http://giphy.com/gifs/zhiliao2-3o85xLfqj5TBqDXzuo

Slide 26

Slide 26 text

http://typicalhope.tumblr.com/