Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Magento go fast

Thijs Feryn
November 12, 2014

Making Magento go fast

My Magento talk at PHP World 2014 in Washington DC

Thijs Feryn

November 12, 2014
Tweet

More Decks by Thijs Feryn

Other Decks in Technology

Transcript

  1. First attempts ✓Install powerful machines ✓Lots of RAM & CPU

    ✓Loadbalancing on the webservers ✓Master/master replication on the MySQL servers ✓High memory limit ✓APC byte code caching
  2. First Magento skills ✓Activate caching in the admin panel ✓Flat

    catalogs ✓The Magento compiler ✓JS & CSS minification
  3. First conclusions ✓It uses a lot of RAM & CPU

    ✓And a whole lot of I/O ✓The MySQL server gets hammered
  4. ./mage-­‐-­‐c   ./mage-­‐-­‐a   ./mage-­‐-­‐4   ./mage-­‐-­‐7   ./mage-­‐-­‐e  

    ./mage-­‐-­‐5   ./mage-­‐-­‐b   ./mage-­‐-­‐9   ./mage-­‐-­‐0   ./mage-­‐-­‐f   ./mage-­‐-­‐3   ./mage-­‐-­‐6   ./mage-­‐-­‐8   ./mage-­‐-­‐d   ./mage-­‐-­‐1
  5. ./mage-­‐-­‐8/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_STORE_BE_EN_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_STORE_LU_EN_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_NL_NL_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_LU_EN_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_LU_FR_CONFIG_CACHE  

    ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_FR_EN_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_STORE_LU_FR_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_BE_FR_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_STORE_ADMIN_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_STORE_BE_FR_CONFIG_CACHE   ./mage-­‐-­‐8/mage-­‐-­‐-­‐MAGE_DB_PDO_MYSQL_DDL_sales_flat_shipment_grid_3   ./mage-­‐-­‐d/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_APP_E4D52B98688947405EDE639E947EE03D   ./mage-­‐-­‐d/mage-­‐-­‐-­‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C   ./mage-­‐-­‐d/mage-­‐-­‐-­‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1   ./mage-­‐-­‐d/mage-­‐-­‐-­‐MAGE_APP_E4D52B98688947405EDE639E947EE03D   ./mage-­‐-­‐d/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C   ./mage-­‐-­‐d/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1   ./mage-­‐-­‐1/mage-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_   ./mage-­‐-­‐1/mage-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_   ./mage-­‐-­‐1/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_currencynumber_   ./mage-­‐-­‐1/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_   ./mage-­‐-­‐1/mage-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_currencynumber_   ./mage-­‐-­‐1/mage-­‐-­‐-­‐internal-­‐metadatas-­‐-­‐-­‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_
  6. APC

  7. In app/etc/local.xml <global>          <cache>    

                 <backend>apc</backend>                  <prefix>MAGE_</prefix>          </cache>   </global>
  8. <global>    <cache>        <memcached>      

       <servers>              <server>                <host><![CDATA[127.0.0.1]]></host>                <port><![CDATA[11211]]></port>                <persistent><![CDATA[1]]></persistent>              </server>          </servers>      </memcached>    </cache>   </global> In app/etc/local.xml Multiple servers
  9. But we made it work on 5.4. (with a patch)

    http://www.magentocommerce.com/knowledge-base/ entry/php54-patch
  10. 19280  33686  1.3    0.4  322052  110408  ?  S  14:07

     0:01  /usr/local/php-­‐5.4/bin/php-­‐cgi   18834  33696  0.0    0.0  267780  13608    ?  S  14:07  0:00  /usr/local/php-­‐5.4/bin/php-­‐cgi   19280  33698  0.9    0.4  321808  110948  ?  S  14:07  0:01  /usr/local/php-­‐5.4/bin/php-­‐cgi   19057  33700  0.0    0.0  268124  13416    ?  S  14:07  0:00  /usr/local/php-­‐5.4/bin/php-­‐cgi   18767  33710  0.0    0.0  269728  14156    ?  S  14:08  0:00  /usr/local/php-­‐5.4/bin/php-­‐cgi Separate processes, no joint cache
  11. root  26067  0.0  0.0  342892  6248  ?  S  13:43  0:00

     php-­‐fpm:  master  process  (/etc/php5/fpm/php-­‐fpm.conf)   www-­‐data  26068    0.2    1.1  396952  94684  ?  S  13:43  0:03  php-­‐fpm:  pool  www   www-­‐data  26069    0.0    0.6  363472  56456  ?  S  13:43  0:01  php-­‐fpm:  pool  www   www-­‐data  26087    0.0    0.6  362708  50188  ?  S  13:44  0:00  php-­‐fpm:  pool  www Master proces that stores the byte code cache
  12. <global>      <resources>          <default_setup>  

               <connection>                  <host><![CDATA[Master-­‐host]]></host>                  <username><![CDATA[user]]></username>                  <password><![CDATA[pass]]></password>                  <dbname><![CDATA[magentodb]]></dbname>                  <active>1</active>              </connection>          </default_setup>          <default_read>              <connection>                  <use/>                  <host><![CDATA[Slave-­‐host]]></host>                  <username><![CDATA[user]]></username>                  <password><![CDATA[pass]]></password>                  <dbname><![CDATA[magento]]></dbname>                  <type>pdo_mysql</type>                  <model>mysql4</model>                  <initStatements>SET  NAMES  utf8</initStatements>                  <active>1</active>              </connection>          </default_read>        </resources>   </global> R/W   splitting
  13. is GREAT! ✓Built-in replication ✓Data types (hash, list, …) ✓Save

    on disk ✓Session clustering ✓Cm_Cache_Backend_Redis by default in Magento 1.8 CE & 1.13 EE ✓No 2-level cache ✓Multiple databases ✓Authentication
  14. <global>      <cache>          <backend>Cm_Cache_Backend_Redis</backend>  

           <backend_options>              <server>127.0.0.1</server>              <port>6379</port>              <persistent></persistent>              <database>1</database>              <password></password>              <force_standalone>0</force_standalone>              <connect_retries>1</connect_retries>              <read_timeout>10</read_timeout>              <automatic_cleaning_factor>0</automatic_cleaning_factor>              <compress_data>1</compress_data>              <compress_tags>1</compress_tags>              <compress_threshold>20480</compress_threshold>              <compression_lib>gzip</compression_lib>          </backend_options>      </cache>   </global> In app/etc/local.xml
  15. In app/etc/modules/Cm_RedisSession.xml <config>      <modules>        

     <Cm_RedisSession>              <active>true</active>              <codePool>community</codePool>          </Cm_RedisSession>      </modules>   </config>
  16. <global>      <session_save>db</session_save>      <redis_session>      

       <host>127.0.0.1</host>          <port>6379</port>          <password></password>          <timeout>2.5</timeout>          <persistent></persistent>          <db>1</db>          <compression_threshold>2048</compression_threshold>          <compression_lib>gzip</compression_lib>          <log_level>1</log_level>          <max_concurrency>6</max_concurrency>          <break_after_frontend>5</break_after_frontend>          <break_after_adminhtml>30</break_after_adminhtml>          <bot_lifetime>7200</bot_lifetime>      </redis_session>   </global> In app/etc/local.xml
  17. Varnish ✓Only works on product catalog & CMS pages ✓Doesn’t

    work on checkout ✓Great for static content ✓No SSL support ✓Best way to cache
  18. Turpentine ✓Community module for Varnish in Magento ✓ESI or AJAX

    for session data ✓Configurable in the admin ✓VCL is pushed via Turpentine to the Varnish over Telnet ✓Takes care of purging ✓SSL termination via Nginx
  19. <global>      <fpc>          <lifetime>86400</lifetime>  

      <debug>false</debug>          <backend>Cm_Cache_Backend_Redis</backend>          <backend_options>              <server>127.0.0.1</server>              <port>6379</port>              <persistent>cache-­‐fpc</persistent>              <database>1</database>              <password></password>              <force_standalone>1</force_standalone>              <connect_retries>1</connect_retries>              <lifetimelimit>86400</lifetimelimit>              <read_timeout>10</read_timeout>              <compress_data>1</compress_data>              <compress_tags>1</compress_tags>              <compress_data>gzip</compress_data>          </backend_options>      </fpc>   </global> In app/etc/local.xml Also  works   with  other   backends
  20. ALTER  TABLE  `log_customer`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_quote`  ENGINE=BLACKHOLE;  

    ALTER  TABLE  `log_summary`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_summary_type`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_url`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_url_info`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_visitor`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_visitor_info`  ENGINE=BLACKHOLE;   ALTER  TABLE  `log_visitor_online`  ENGINE=BLACKHOLE;
  21. Loadbalanced ElasticSearch Redis Loadbalanced MySQL MySQL Loadbalanced frontend Nginx Nginx

    Loadbalanced Varnish Varnish Loadbalanced backend Nginx Nginx Client Admin PHP-FPM PHP-FPM PHP-FPM PHP-FPM SSL endpoint ElasticSearch
  22. Redis “clustering” ✓Standalone Redis instance per webserver ✓Twemproxy & Twemproxy

    agent ✓Redis-cluster (in v3, which is not yet stable) ✓Redis Sentinel & keepalived ✓Redis Sentinel patch for Cm_Cache_Backend_Redis
  23. Sentinel ✓Discovery service for Redis servers ✓Part of the Redis

    core ✓Monitoring, notification & autofailover ✓Knows which are masters & slaves ✓Avoid problems when the master is down ✓Avoids problems when to master is up again ✓First connect to Sentinel, then to Redis
  24. port  26379   sentinel  monitor  mymaster  127.0.0.1  6379  2  

    sentinel  down-­‐after-­‐milliseconds  mymaster  5000   sentinel  parallel-­‐syncs  mymaster  2   sentinel  client-­‐reconfig-­‐script  mymaster  ./sentinel.sh Sentinel config
  25. $>  redis-­‐cli  -­‐p  26379   127.0.0.1:26379>  sentinel  master  mymaster  

     1)  "name"    2)  "mymaster"    3)  "ip"    4)  "127.0.0.1"    5)  "port"    6)  "6379"    7)  "runid"    8)  "e3a3fbab9687857c74266b360a85317aea2a213c"    9)  "flags"   10)  "master"   11)  "pending-­‐commands"   12)  "0"   13)  "last-­‐ping-­‐sent"   14)  "0"   15)  "last-­‐ok-­‐ping-­‐reply"   16)  "268"   17)  "last-­‐ping-­‐reply"   Discover master
  26. $>  redis-­‐cli  -­‐p  26379   127.0.0.1:26379>  sentinel  slave  mymaster  

    1)    1)  "name"          2)  "127.0.0.1:6380"          3)  "ip"          4)  "127.0.0.1"          5)  "port"          6)  "6380"          7)  "runid"          8)  "40a20d214c30b5a63a78882ee68d51f2b8506cc0"          9)  "flags"        10)  "slave"        11)  "pending-­‐commands"        12)  "0"        13)  "last-­‐ping-­‐sent"        14)  "0"        15)  "last-­‐ok-­‐ping-­‐reply"        16)  "158"        17)  "last-­‐ping-­‐reply"        18)  "158"   Discover slaves Supports multiple slaves
  27. Keepalived vrrp_script  chk_redis_master  {          script  "/vagrant/keepalived.sh"

             interval  2   }   vrrp_instance  redis  {          state  MASTER          interface  eth1          virtual_router_id  51          priority  150          advert_int  1          authentication  {                  auth_type  PASS                  auth_pass  thepasswordispassword          }          virtual_ipaddress  {                  10.10.10.42          }          track_script  {                  chk_redis_master          }           } vrrp_script  chk_redis_master  {          script  "/vagrant/keepalived.sh"          interval  2   }   vrrp_instance  redis  {          state  MASTER          interface  eth1          virtual_router_id  51          priority  100          advert_int  1          authentication  {                  auth_type  PASS                  auth_pass  thepasswordispassword          }          virtual_ipaddress  {                  10.10.10.42          }          track_script  {                  chk_redis_master          }           } Master Slave
  28. Keepalived #!/bin/bash   result=$(redis-­‐cli  -­‐p  26379  SENTINEL  get-­‐master-­‐addr-­‐by-­‐name  mymaster  |

     head  -­‐n1  2>&1)   ip=`/sbin/ifconfig  eth1  |  grep  'inet  addr:'  |  cut  -­‐d:  -­‐f2  |  awk  '{  print  $1}'`   if  [  "$result"  ==  "$ip"  ]  ;   then      echo  "I'm  master"      exit  0   else      echo  "I'm  slave"      exit  1   fi check script
  29. Twemproxy ✓Redis & Memcached Proxy ✓Written by Twitter ✓Does sharding

    ✓Persistent connections ✓Disable nodes on failure https://github.com/twitter/twemproxy
  30. Twemproxy twem1:      listen:  127.0.0.1:22122      hash:  fnv1a_64

         hash_tag:  "{}"      distribution:  ketama      auto_eject_hosts:  false      timeout:  400      redis:  true      servers:        -­‐  10.10.10.40:6379:1  master1        -­‐  10.10.10.43:6379:1  master2
  31. Twemproxy agent ✓NodeJS daemon ✓Monitors master status via Sentinel ✓Changes

    Twemproxy config on failover https://github.com/Stono/redis-twemproxy-agent
  32. Twemproxy agent node  cli.js  -­‐h  10.10.10.40  -­‐p  26379  \  

    -­‐f  /etc/twemproxy/conf/magento.conf  \   -­‐c  /etc/init.d/twemproxy  restart  \   -­‐l  /var/log/twemproxy/twemproxy_sentinel.log
  33. port  7000   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout

     5000   appendonly  yes port  7001   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout  5000   appendonly  yes port  7002   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout  5000   appendonly  yes port  7003   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout  5000   appendonly  yes port  7004   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout  5000   appendonly  yes port  7005   cluster-­‐enabled  yes   cluster-­‐config-­‐file  nodes.conf   cluster-­‐node-­‐timeout  5000   appendonly  yes Cluster node config
  34. $>  ./redis-­‐trib.rb  create  -­‐-­‐replicas  1  127.0.0.1:7000  \   127.0.0.1:7001  127.0.0.1:7002

     127.0.0.1:7003  127.0.0.1:7004  \   127.0.0.1:7005   >>>  Creating  cluster   Connecting  to  node  127.0.0.1:7000:  OK   Connecting  to  node  127.0.0.1:7001:  OK   Connecting  to  node  127.0.0.1:7002:  OK   Connecting  to  node  127.0.0.1:7003:  OK   Connecting  to  node  127.0.0.1:7004:  OK   Connecting  to  node  127.0.0.1:7005:  OK   >>>  Performing  hash  slots  allocation  on  6  nodes...   Using  3  masters:   127.0.0.1:7000   127.0.0.1:7001   127.0.0.1:7002   Creating the cluster
  35. $>  redis-­‐cli  -­‐c  -­‐p  7000   127.0.0.1:7000>  set  key  value

      -­‐>  Redirected  to  slot  [12539]  located  at   127.0.0.1:7002   OK   127.0.0.1:7000>  get  key   "value"   127.0.0.1:7000> Using the cluster $>  redis-­‐cli  -­‐c  -­‐p  7005   127.0.0.1:7005>  get  key   -­‐>  Redirected  to  slot  [12539]  located  at   127.0.0.1:7002   "value"   127.0.0.1:7005>  get  key   "value"