Troubleshooting Ecommerce Performance

Presented at SuiteWorld 2018 in Las Vegas, Nevada.

Studies show a direct correlation between website speed and conversion rates: faster speeds equals higher revenue and bigger profits. In this session, we'll demonstrate how to detect and correct various performance pitfalls in SuiteCommerce. We'll teach you all about best practices and tools for ensuring that your web store outperforms your competitors by more than 30%.

Transcript

1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 1 Mark Sweeting & Diego Cardozo Troubleshooting Ecommerce Performance April 24, 2018 COBRAND NAME
2. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 3 THE SCENARIO A customer has contacted us stating that they have a performance problem with their website. They provide no details about the nature of the problem.
4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 4 WHERE TO START – ANY METRICS?
5. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 5 STEP 1: CONTENT DELIVERY NETWORK
6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 6 WHAT IS A CONTENT DELIVERY NETWORK?
7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 7 WHAT IS A CONTENT DELIVERY NETWORK?
8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 8 TOOL: DIG > dig IN A www.example.com
9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 9 IS THE CDN CONFIGURED AND WORKING? Mac/Linux users, use dig command to check: > dig IN A www.example.com Windows users use nslookup: > nslookup -class=IN -querytype=A www.example.com • Note: use full hostname without protocol (http or https)
10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 10 IS THE CDN CONFIGURED AND WORKING? GOOD: domain resolves to Akamai A-records www.example.com. 300 IN CNAME www.example.com.hosting.netsuite.com. www.example…netsuite.com. 300 IN CNAME website-cdn.na1.netsuite.com. website-cdn.na1.netsuite.com. 54 IN CNAME website-cdn.na1.netsuite.com.mdc.edgesuite.net. website-cdn.na1.netsuite.com.mdc.edgesuite.net. 415 IN CNAME a1280.q.akamai.net. a1280.q.akamai.net. 14 IN A 213.123.255.6 a1280.q.akamai.net. 14 IN A 213.123.255.9
11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 11 GOOD: domain resolves to Cloudfront A-records www.example.com. 9 IN CNAME www.example.com.hosting.netsuite.com. www.example......netsuite.com. 299 IN CNAME abcdefghijklmn.cloudfront.net. abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.242 abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.3 abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.96 abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.102 abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.166 abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.184 IS THE CDN CONFIGURED AND WORKING?
12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 12 IS THE CDN CONFIGURED AND WORKING? BAD: CNAME OK, but CDN tickbox not checked; resolves to Netsuite datacenter. www.example.com. 600 IN CNAME www.example.com.hosting.netsuite.com. www.example...ting.netsuite.com. 300 IN CNAME 9999999-origin.shop.netsuite.com. 9999999-origin.shop.netsuite.com. 300 IN CNAME shop.source.netsuite.com. shop.source.netsuite.com. 66 IN CNAME shopping.netsuite.com. shopping.netsuite.com. 286 IN A 167.216.129.13
13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 13 IS THE CDN CONFIGURED AND WORKING? BAD: CNAME not created; A record points straight to Netsuite datacenter. www.example.com. 600 IN A 167.216.129.13
14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 14 IS THE CDN CONFIGURED AND WORKING? Use Akamai Debug Headers Chrome extension.
15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 15 STEP 2: SEO PAGE GENERATOR
16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 16 WHAT IS THE SEO PAGE GENERATOR
17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 17 TTFB: TIME TO FIRST BYTE • Time it takes for first byte of response content to reach the client browser • Part of the following sequence: −DNS LOOKUP −TCP CONNECTION −SSL NEGOTIATION −REQUEST SEND −TIME TO FIRST BYTE −CONTENT DOWNLOAD (the HTML) • Usually the time it takes for dynamic server-side activities to happen
18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 18 TIME TO FIRST BYTE
19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 19 CHECK THE SEO PAGE GENERATOR
20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 20 TOOL: PAGE GENERATOR DEBUGGER ?seodebug=T
21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 21 HOW TO DEBUG THE PAGE GENERATOR 1. Turn on the debugger: − seodebug=T 2. Bust the cache: − preview=<random string> 3. Force Pre-render (for additional useful output) − seoprerender=T Example: http://www.example.com/some-page?seodebug=T&preview=12345 => Then view the page source…
22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 22 HOW TO DEBUG THE PAGE GENERATOR • Check the main DIV isn’t empty: <div id=“main”></div> • Search for errors in the output • Examine the timings of sub-requests (prerender only) • Check the status codes of sub-requests (prerender only) • Check sub-requests don’t include any unnecessary content (third parties!)
23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 23 DEBUGGING: CHECK MAIN DIV IS POPULATED
24. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 24 DEBUGGING: SEARCH FOR ERRORS [02:15:04.188] [ 50961 kb ] [ +34 ms ] [ -574 kb ] Envjs.XMLHttpRequest.Jav8 DEBUG: [Fri, 23 Feb 2018 10:15:04 GMT:187] contentEncoding utf-8 [02:15:04.190] [ 51035 kb ] [ +2 ms ] [ +74 kb ] Envjs.Jav8 ERROR: [Fri, 23 Feb 2018 10:15:04 GMT:190] printStackTrace error {anonymous}("java.io.IOException: Server returned HTTP response code: 400 for URL: https://www.example.com/api/items?country=US&currency=USD&custitem_ef_badges=Best Seller&fieldset=details&language=en&limit=12&pricelevel=5"),printStackTrace(#object),{anonymous}("java.i o.IOException: Server returned HTTP response code: 400 for URL: https://www.example.com/api/items?country=US&currency=USD&custitem_ef_badges=Best Seller&fieldset=details&language=en&limit=12&pricelevel=5"),{anonymou [02:15:04.191] [ 51043 kb ] [ +1 ms ] [ +8 kb ] Envjs.XMLHttpRequest.Jav8 DEBUG: [Fri, 23 Feb 2018 10:15:04 GMT:191] Envjs.connection: finished
25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 25 DEBUGGING: SLOW SUB-REQUESTS [07:23:34.334] [ +0 ms ] Sub request total: 4.453000s [07:23:34.335] [ +1 ms ] Details of 16 sub requests: GET http://www.example.com/cms/2/assets/js/postframe.js [status 200] Requested at 2018-02-19T15:23:29.414Z and responded by 2018-02-19T15:23:29.497Z (which took 83ms) GET http://www.example.com/cms/2/cms.js [status 200] Requested at 2018-02-19T15:23:29.414Z and responded by 2018-02-19T15:23:29.618Z (which took 204ms) GET http://www.example.com/c.1234567/ShopFlow/sc.environment.ssp?v=1&lang=en_US&cur=AUD [status 302] Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.614Z (which took 116ms) GET http://www.example.com/c.1234567/ShopFlow/js/libs/Libraries-0145005a1354.js [status 200] Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.617Z (which took 119ms) GET http://www.example.com/c.1234567/ShopFlow/templates/Templates-01405a477c8c.js [status 200] Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.615Z (which took 117ms) GET http://www.example.com/c.1234567/ShopFlow/js/Application-01450087b360.js [status 200] Requested at 2018-02-19T15:23:29.499Z and responded by 2018-02-19T15:23:29.616Z (which took 117ms) GET http://www.example.com/c.1234567/CustomWS/common-shopping-customizations.js?v=80 [status 200] Requested at 2018-02-19T15:23:29.499Z and responded by 2018-02-19T15:23:29.617Z (which took 118ms) GET http://www.example.com/c.1234567/ShopFlow/languages/en_US.js?n=18 [status 200] Requested at 2018-02-19T15:23:29.500Z and responded by 2018-02-19T15:23:29.796Z (which took 296ms) GET http://www.example.com/c.1234567/ShopFlow/sc.environment.ssp?cur=AUD&lang=en_US&v=1&whence= [status 200] Requested at 2018-02-19T15:23:29.614Z and responded by 2018-02-19T15:23:32.624Z (which took 3010ms) GET http://www.example.com/api/cms/session/domain [status 200] Requested at 2018-02-19T15:23:32.838Z and responded by 2018-02-19T15:23:32.947Z (which took 109ms) GET http://www.example.com/c.1234567/ShopFlow/services/live-order.ss?internalid=cart&t=1519053812858 [status 200] Requested at 2018-02-19T15:23:32.858Z and responded by 2018-02-19T15:23:33.147Z (which took 289ms) GET http://www.example.com/c.1234567/cds/services/page.ss?internalid=25&ttl=500&cache=MEDIUM [status 200] Requested at 2018-02-19T15:23:32.882Z and responded by 2018-02-19T15:23:33.367Z (which took 485ms) GET http://www.example.com/api/cms/versions?site_id=16&c.1234567 [status 200] Requested at 2018-02-19T15:23:32.948Z and responded by 2018-02-19T15:23:33.047Z (which took 99ms) GET http://www.example.com/api/cms/pages/contents?version_id=2815&site_id=16&c.1234567&path=%2F&page_type= [status 200] Requested at 2018-02-19T15:23:33.049Z and responded by 2018-02-19T15:23:33.247Z (which took 198ms) GET http://www.example.com/c.1234567/CustomWS/BannerSlider/shopping/services/getSlides.ss [status 200] Requested at 2018-02-19T15:23:33.394Z and responded by 2018-02-19T15:23:33.650Z (which took 256ms) GET http://www.example.com/api/cms/pages/contents?version_id=2815&site_id=16&c.1234567&path=%2F&page_type=home-page [status 200] Requested at 2018-02-19T15:23:33.399Z and responded by 2018-02-19T15:23:33.549Z (which took 150ms)
26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 26 DEBUGGING: FAILED SUB-REQUESTS [07:49:18.431] [ +0 ms ] Sub request total: 0.414000s [07:49:18.431] [ +0 ms ] Details of 4 sub requests: GET https://www.example.com/c.1234567/sca-dev-elbrus/shopping.environment.ssp?lang=en_US&cur=null [status 502] Requested at 2018-02-19T15:49:17.415Z and responded by 2018-02-19T15:49:17.502Z (which took 87ms) GET https://www.example.com/c.1234567/sca-dev-elbrus/languages/shopping_en_US.js?t=1516304321167 [status 502] Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.613Z (which took 197ms) GET https://www.example.com/c.1234567/sca-dev-elbrus/javascript/shopping.js?t=1516304321167 [status 502] Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.615Z (which took 199ms) GET https://www.example.com/cms/2/cms.js [status 502] Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.615Z (which took 199ms)
27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 27 STEP 3: OPTIMIZING CONTENT
28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 28 TOOL: WEBPAGE TEST http://www.webpagetest.org/
29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 29 USING WEB PAGETEST
30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 30 WEBPAGE TEST WATERFALL – 161 REQUESTS
31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 31 WEBPAGE TEST – BLOCKING RESPONSES
32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 32 WEBPAGE TEST – CONTENT BREAKDOWN
33. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 33 IMAGE SIZES • Get the image sizes rights • Use the image resizing service
34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 34 IMAGE SIZES • Get the image sizes rights • Use the image resizing service
35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 35 TOOL: IMAGEOPTIM https://imageoptim.com/
36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 36 IMAGEOPTIM https://imageoptim.com/mac
37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 37 STEP 4: PAGE RENDERING
38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 38 RENDERING TIMELINE – WEB PAGETEST
39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 39 PAGE RENDERING – THE USUAL SUSPECTS • Slow blocking responses (i.e. blocking parallel downloads) • Fonts – can seriously delay page rendering • Third parties – insert blocking JavaScript and also delay page rendering − WebPage Test can give you an overview of the percentage of requests that correspond to third parties (domain breakdown) − You can test the impact of removing third parties by right clicking their requests on Chrome’s Dev Tools and selecting “Block request domain”
40. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 40 SLOW SUB-REQUESTS • By this stage we’ve ruled out CDN, page generator, and content weight as causing problems, but there are many other possible causes of slow page load. • We may have identified some of these problems when looking at the waterfall in Webpage Test, or when looking at the sub- requests log in the SEO Debug output, for example: −Item API −Environment files −Categories −Third parties
41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 41 STEP 5: ITEM API
42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 42 ITEM API • We need to understand how search works in order to understand why it can be slow • A SolR query is ran on the application server, which indicates which items are to be returned • A database query is ran on the database server, which returns data for those items • We can see those timings the timings using the following parameter: &ssdebug=T Browser SolR Database query
43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 43 ITEM API 1. Open request on chrome dev tools 2. Add &ssdebug=T parameter SQL = 2.5 s
44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 44 ITEM API Item (24) Item Data Related Items Custom Fields Item Data, Stock Info, Images • itemid • internalid • displayname • ispurchasable • showoutofstockmessage • stockdescription • outofstockmessage • isbackorderable • isinstock Stock Info Images Pricing Info • onlinecustomerprice • onlinecustomerprice_formatted • priceschedule
45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 45 ITEM API Item (24) Item Data Related Items Custom Fields • ispurchasable • showoutofstockmessage • stockdescription • outofstockmessage • isbackorderable • isinstock Stock Info Images Pricing Info • onlinecustomerprice • onlinecustomerprice_formatted • priceschedule
46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 46 STEP 6: ENVIRONMENT FILES
47. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 47 ENVIRONMENT FILES – USER ENVIRONMENT •shopping.user.environment is blocking but non-cacheable •Only user-specific content should be found here •Troubleshooting − Check for any information that is not user-specific and remove it − Developers sometimes place data here to make sure that it is always fresh − The problem is that it is loaded in all pages
48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 48 ENVIRONMENT FILES – REGULAR ENVIRONMENT •shopping.environment.ssp is blocking but cacheable •You can bootstrap information here, but only if it is needed on all pages •If the information is not needed on all pages, you can implement a service to retrieve it on demand •shopping.environment.ssp usually includes: − Categories − CMS − Configuration
49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 49 STEP 7: CATEGORIES
50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 50 CATEGORIES •Design your category structure for performance and usability •3 golden rules for blazingly fast categories: −Use commerce categories −Your category tree shouldn’t be deeper than 3 levels −Each product should be assigned to a single category
51. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 51 CATEGORIES
52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 52 STEP 8: SCRIPTABLE CART
53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 53 SCRIPTABLE CART •What is scriptable cart? •Parallelism between the back end and the web store Sales Order form  BOGO
54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 54 SCRIPTABLE CART •What is scriptable cart? −Executes client scripts for the sales order record on the web store −Per-website setting −Problem: it executes ALL client scripts •If adding items to the cart is slow, this is generally why
55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 55 SCRIPTABLE CART - TROUBLESHOOTING •Customization  Scripting  Scripted Records
56. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 56 SCRIPTABLE CART - TROUBLESHOOTING •Solution: exclude client scripts from the web store −If you set execution context to “All Roles”  it executes on the web store −If you set it to “All Employees”  it doesn’t
57. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 57 STEP 9: USER EVENTS
58. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 58 PLACE ORDER •Issues on order placement times usually come from workflows and User Events •User Event script execution context −Checkbox called “Execute in Commerce Context” within the script’s deployment −When this option is not selected, the script will not trigger for events that take place in SCA, SB or SCIS
59. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 59 PLACE ORDER – USER EVENT EXECUTION CONTEXT
60. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 60 PLACE ORDER •Setup  Company  Enable Features  Web Presence −Asynchronous AfterSubmit Sales Order Processing
61. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 61 TOOL: NETSUITE APM
62. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 62 NETSUITE APM
63. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 63 NETSUITE APM
64. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 64 NETSUITE APM •It only records back end activity •These are the same scripts that run on the web store •You need to know which scripts execute on the web −Client scripts: will run for the shopper role −User Events: if “execute in commerce” is enabled
65. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 65 IN A NUTSHELL
66. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 66 STEP-BY-STEP RECAP 1. Check the Content Delivery Network (CDN) 2. Check the SEO Page Generator 3. Optimize your content (use WebPage Test and imageoptim) 4. Optimize page rendering (fonts, third parties, blocking requests) 5. Optimize the Item API 6. Check environment files 7. Review your category structure for speed 8. Review scripts that impact cart timings (scriptable cart) 9. Review user events (use APM) SCA SC SB
67. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 67 NEED SOME HELP? WE ARE THE 10th STEP
68. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 68 PERFORMANCE SERVICES If you still need some help with investigating or remedying performance issues, then we can help. • Performance reports and benchmarking • Monitoring and alerting • Performance audits • Built to Perform (for new projects) diego.cardozo@oracle.com mark.sweeting@oracle.com
69. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 69 Q&A
70. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 70
71. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 71 Get involved, visit on Level 3 SuiteWorld
72. Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 72 THANK YOU