toom + Magento - a journey to zero downtime (Munich Meetup)

Transcript

1. a journey to zero-downtime + +
3. Annual turnover of ~€2.6 billion
4. Around 350 markets
5. Technical Challenges
6. Some of our challenges • Market specific pricing and stock • Huge volumes of stock changes • Huge catalog provided by a middleware • Zero-downtime deployments
7. Problem: Market specific pricing and stock
8. 350+ markets How do we model these in Magento?
9. Magento stores? Increased load time, bigger indexes
10. Solution: custom Market model Bespoke stock / price model too
12. Replace Magento’s stock registry
13. Replace Magento’s price pool types
14. Custom price + stock indexes Increased runtime performance
15. Go microservices for stock and price
16. stockSales Kafka Microservice stock-service Microservice stock-change- import Shop Magento 2.2 MySQL Magento DB
17. Kafka provides real-time updates Stock and price changes as they happen
18. Problem: Millions of stock updates weekly How do we keep our indexes up-to-date?
19. Solution: Index updates with MView
20. Building huge indexes Tens of millions of entries
21. Problem: Large query results loaded into PHP memory by default
22. “Queries are using the buffered mode by default. This means that query results are immediately transferred from the MySQL Server to PHP and then are kept in the memory of the PHP process 
 http://php.net/manual/en/mysqlinfo.concepts.buffering.php
23. Solution: Unbuffered queries ‘use_buffered_query’ => false
24. “Unbuffered MySQL queries execute the query and then return a resource while the data is still waiting on the MySQL server for being fetched. This uses less memory on the PHP-side, but can increase the load on the server.”
 
 http://php.net/manual/en/mysqlinfo.concepts.buffering.php
25. $conn = $resourceConnection->getConnection($name);
 $config = $conn->getConfig();
 $config['use_buffered_query'] = false;
 
 $unbufferedConn = $objectManager->create( get_class($conn), ['config' => $config] );
26. MySQL server now buffers the result Much smaller memory footprint in PHP process
27. Problem: MView was too slow Out of date stock information
29. Solution: Stock changes via RabbitMQ
30. We moved stock data out of Magento DB
31. stockSales Kafka Microservice stock-service Microservice stock-change- import Shop Magento 2.2 MySQL Stock DB MySQL Magento DB
32. stockSales Kafka Microservice stock-service Microservice stock-change- import Shop stock-change-consumer MySQL Stock DB MySQL Magento DB RabbitMQ stock-changes Shop stock-change-consumer
33. Lesson: Indexes are powerful, use them Partial index updates are great, but benchmark for your use case
34. Lesson: Profile your solutions https://tideways.com/ or https://blackfire.io/
35. Problem: Huge product catalog 100,000+ products, 2400+ attributes, 950+ sets
36. Middleware receives data from PIM
37. Magento REST API Middleware S4T Connect POST /rest/V1/products HTTP 200 Success MySQL Magento DB Product Data PIM
38. Slow full catalog import Needed to improve this
39. Solution: parallelise the REST API RabbitMQ + background processing
40. Request is stored in RabbitMQ HTTP 202 Accepted response code
41. Magento queue processor RabbitMQ catalog-requests Magento queue processor Magento queue processor Magento REST API Middleware S4T Connect POST /rest/V1/products HTTP 202 Accepted MySQL Magento DB
42. API response time decreased 850% response time improvement
43. Product creation time improved 6x faster to persist 1000 products in the catalog
44. Lesson: Think outside the box Magento provides us with the tools to create tailored solutions
45. Problem: Zero-downtime deployments Maintenance page free deploys
46. Database migrations Automated database updates executed during the deployment
47. Native Magento database migrations module.xml and setup_version
 (2.2.x)
48. Module version mismatches Magento will return a 5xx response
49. Maintenance page = downtime Breaks zero-downtime for toom
50. Our solution: Phinx migrations http://phinx.org/ and Magento ❤
51. Provides new trigger for DB changes No need to change setup_version in module.xml
52. 1. Run bin/magento setup:db:status 2. If exit code is 2, run bin/magento setup:upgrade 3. Run bin/phinx migrate 4. Continue as normal New database migration process for deploys
53. Make DB changes backwards compatible Should work with both old and new code
54. What about declarative schema in 2.3?
55. Magento applies schema XML in code to the DB
56. Another problem: shared cache data During deploy both old and new code can be hit by web traffic
57. Request HTTP Magento New Code Magento Old Code Magento New Code Magento Old Code Redis Cache DB
58. Solution: Alternate Redis database Should work with both old and new code
59. Request HTTP Magento New Code Magento Old Code Magento New Code Magento Old Code Redis New DB Old DB
60. Zero Downtime
61. Lesson: Try and avoid setup:upgrade on every deployment
62. Lesson: Try out declarative schema
63. Key take-aways What did we learn from toom + Magento?
64. 1. Big customisations are achievable
65. 2. Parallelise big tasks with RabbitMQ
66. 3. Custom indexes are powerful
67. 4. Always profile your solutions
68. 5. Zero-downtime deploys are possible
69. I’m James Halsall @jameshalsall
70. Thanks