Making Magento go fast - Speaker Deck

Making Magento go fast

by Thijs Feryn

Slide 1

Slide 1 text

Making Magento go fast By Thijs Feryn

Slide 2

Slide 2 text

my name Thijs Hi, is

Slide 3

Slide 3 text

And I’m @ThijsFeryn on Twitter

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

I’m an at Evangelist

Slide 6

Slide 6 text

I’m at a board member

Slide 7

Slide 7 text

https://joind.in/talk/view/11867 Feedback

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Magento is kinda slow …

Slide 10

Slide 10 text

But tremendously flexible

Slide 11

Slide 11 text

In the beginning we were somewhat clueless

Slide 12

Slide 12 text

But we learned a lot!

Slide 13

Slide 13 text

It did take some time though

Slide 14

Slide 14 text

Operations approach

Slide 15

Slide 15 text

The first attempts

Slide 16

Slide 16 text

First attempts ✓Install powerful machines ✓Lots of RAM & CPU ✓Loadbalancing on the webservers ✓Master/master replication on the MySQL servers ✓High memory limit ✓APC byte code caching

Slide 17

Slide 17 text

Looking for Magento knowledge

Slide 18

Slide 18 text

First Magento skills ✓Activate caching in the admin panel ✓Flat catalogs ✓The Magento compiler ✓JS & CSS minification

Slide 19

Slide 19 text

The Magento compiler is a lie!

Slide 20

Slide 20 text

Use a real byte code cache instead

Slide 21

Slide 21 text

Things started to speed up But it wasn’t fantastic

Slide 22

Slide 22 text

First conclusions

Slide 23

Slide 23 text

First conclusions ✓It uses a lot of RAM & CPU ✓And a whole lot of I/O ✓The MySQL server gets hammered

Slide 24

Slide 24 text

We learned how to cache

Slide 25

Slide 25 text

By default on disk in in /var/cache

Slide 26

Slide 26 text

./mage-‐-‐c ./mage-‐-‐a ./mage-‐-‐4 ./mage-‐-‐7 ./mage-‐-‐e ./mage-‐-‐5 ./mage-‐-‐b ./mage-‐-‐9 ./mage-‐-‐0 ./mage-‐-‐f ./mage-‐-‐3 ./mage-‐-‐6 ./mage-‐-‐8 ./mage-‐-‐d ./mage-‐-‐1

Slide 27

Slide 27 text

./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_NL_NL_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_FR_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_ADMIN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_sales_flat_shipment_grid_3 ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_

Slide 28

Slide 28 text

APC

Slide 29

Slide 29 text

In app/etc/local.xml apc MAGE_

Slide 30

Slide 30 text

Memcached

Slide 31

Slide 31 text

In app/etc/local.xml Multiple servers

Slide 32

Slide 32 text

Still there are files in /var/cache

Slide 33

Slide 33 text

2-level cache ✓Fast backend first (adapter) ✓Then the slow backend (disk)

Slide 34

Slide 34 text

/var/cache in tmpfs mount -‐t tmpfs -‐o size=20m tmpfs /var/www/var/cache

Slide 35

Slide 35 text

Bigger machines

Slide 36

Slide 36 text

is slow

Slide 37

Slide 37 text

Especially the older versions

Slide 38

Slide 38 text

Magento doesn’t officially work on the latest versions

Slide 39

Slide 39 text

But we made it work on 5.4. (with a patch) http://www.magentocommerce.com/knowledge-base/ entry/php54-patch

Slide 40

Slide 40 text

Without byte code caching, you’re in trouble

Slide 41

Slide 41 text

We forgot that byte code caches aren’t efficient on FastCGI

Slide 42

Slide 42 text

19280 33686 1.3 0.4 322052 110408 ? S 14:07 0:01 /usr/local/php-‐5.4/bin/php-‐cgi 18834 33696 0.0 0.0 267780 13608 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 19280 33698 0.9 0.4 321808 110948 ? S 14:07 0:01 /usr/local/php-‐5.4/bin/php-‐cgi 19057 33700 0.0 0.0 268124 13416 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 18767 33710 0.0 0.0 269728 14156 ? S 14:08 0:00 /usr/local/php-‐5.4/bin/php-‐cgi Separate processes, no joint cache

Slide 43

Slide 43 text

And we forgot that APC is slowly dying

Slide 44

Slide 44 text

-FPM

Slide 45

Slide 45 text

root 26067 0.0 0.0 342892 6248 ? S 13:43 0:00 php-‐fpm: master process (/etc/php5/fpm/php-‐fpm.conf) www-‐data 26068 0.2 1.1 396952 94684 ? S 13:43 0:03 php-‐fpm: pool www www-‐data 26069 0.0 0.6 363472 56456 ? S 13:43 0:01 php-‐fpm: pool www www-‐data 26087 0.0 0.6 362708 50188 ? S 13:44 0:00 php-‐fpm: pool www Master proces that stores the byte code cache

Slide 46

Slide 46 text

Opcache is awesome … and a lot faster

Slide 47

Slide 47 text

Everything in memory!!!

Slide 48

Slide 48 text

We looked at HHVM

Slide 49

Slide 49 text

But the database is also important

Slide 50

Slide 50 text

innodb_buffer_pool_size = 1024M Give MySQL more RAM

Slide 51

Slide 51 text

Turn off query caching

Slide 52

Slide 52 text

innodb_flush_log_at_trx_commit

Slide 53

Slide 53 text

Which one will we choose?

Slide 54

Slide 54 text

1 pdo_mysql mysql4 SET NAMES utf8 1 R/W splitting

Slide 55

Slide 55 text

And all of a sudden, we discovered Redis!

Slide 56

Slide 56 text

is GREAT! ✓Built-in replication ✓Data types (hash, list, …) ✓Save on disk ✓Session clustering ✓Cm_Cache_Backend_Redis by default in Magento 1.8 CE & 1.13 EE ✓No 2-level cache ✓Multiple databases ✓Authentication

Slide 57

Slide 57 text

Cm_Cache_Backend_Redis https://github.com/colinmollenhour/ Cm_Cache_Backend_Redis

Slide 58

Slide 58 text

Cm_Cache_Backend_Redis 127.0.0.1 6379 1 0 1 10 0 1 1 20480 gzip In app/etc/local.xml

Slide 59

Slide 59 text

Cm_RedisSession https://github.com/colinmollenhour/Cm_RedisSession

Slide 60

Slide 60 text

In app/etc/modules/Cm_RedisSession.xml true community

Slide 61

Slide 61 text

db 127.0.0.1 6379 2.5 1 2048 gzip 1 6 5 30 7200 In app/etc/local.xml

Slide 62

Slide 62 text

Varnish

Slide 63

Slide 63 text

Varnish ✓Only works on product catalog & CMS pages ✓Doesn’t work on checkout ✓Great for static content ✓No SSL support ✓Best way to cache

Slide 64

Slide 64 text

Hit rate kinda sucked

Slide 65

Slide 65 text

Turpentine https://github.com/nexcess/magento-turpentine

Slide 66

Slide 66 text

Turpentine ✓Community module for Varnish in Magento ✓ESI or AJAX for session data ✓Configurable in the admin ✓VCL is pushed via Turpentine to the Varnish over Telnet ✓Takes care of purging ✓SSL termination via Nginx

Slide 67

Slide 67 text

Lesti_FPC When Varnish is not an option https://github.com/GordonLesti/Lesti_Fpc

Slide 68

Slide 68 text

86400 false Cm_Cache_Backend_Redis 127.0.0.1 6379 cache-‐fpc 1 1 1 86400 10 1 1 gzip In app/etc/local.xml Also works with other backends

Slide 69

Slide 69 text

Turn your log tables into blackhole tables You’re not using them when you have Varnish/ Lesti_FPC

Slide 70

Slide 70 text

ALTER TABLE `log_customer` ENGINE=BLACKHOLE; ALTER TABLE `log_quote` ENGINE=BLACKHOLE; ALTER TABLE `log_summary` ENGINE=BLACKHOLE; ALTER TABLE `log_summary_type` ENGINE=BLACKHOLE; ALTER TABLE `log_url` ENGINE=BLACKHOLE; ALTER TABLE `log_url_info` ENGINE=BLACKHOLE; ALTER TABLE `log_visitor` ENGINE=BLACKHOLE; ALTER TABLE `log_visitor_info` ENGINE=BLACKHOLE; ALTER TABLE `log_visitor_online` ENGINE=BLACKHOLE;

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Full-text search on the database is stupid … and slow …

Slide 73

Slide 73 text

You can take down certain Magento shops by doing excessive search

Slide 74

Slide 74 text

You could use

Slide 75

Slide 75 text

But I prefer https://github.com/jreinke/magento-Elasticsearch

Slide 76

Slide 76 text

Host static files separately (~CDN) http://www.magentocommerce.com/magento-connect/onepica- imagecdn-1.html

Slide 77

Slide 77 text

Horizontal scalability

Slide 78

Slide 78 text

Loadbalanced ElasticSearch Redis Loadbalanced MySQL MySQL Loadbalanced frontend Nginx Nginx Loadbalanced Varnish Varnish Loadbalanced backend Nginx Nginx Client Admin PHP-FPM PHP-FPM PHP-FPM PHP-FPM SSL endpoint ElasticSearch

Slide 79

Slide 79 text

Problem!

Slide 80

Slide 80 text

Cm_Cache_Backend_Redis can’t handle multiple instances

Slide 81

Slide 81 text

Got that covered!

Slide 82

Slide 82 text

Redis “clustering” ✓Standalone Redis instance per webserver ✓Twemproxy & Twemproxy agent ✓Redis-cluster (in v3, which is not yet stable) ✓Redis Sentinel & keepalived ✓Redis Sentinel patch for Cm_Cache_Backend_Redis

Slide 83

Slide 83 text

Sentinel

Slide 84

Slide 84 text

Sentinel ✓Discovery service for Redis servers ✓Part of the Redis core ✓Monitoring, notification & autofailover ✓Knows which are masters & slaves ✓Avoid problems when the master is down ✓Avoids problems when to master is up again ✓First connect to Sentinel, then to Redis

Slide 85

Slide 85 text

Loadbalanced Redis master Client Loadbalanced Sentinel Sentinel Redis slave Redis slave

Slide 86

Slide 86 text

port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-‐after-‐milliseconds mymaster 5000 sentinel parallel-‐syncs mymaster 2 sentinel client-‐reconfig-‐script mymaster ./sentinel.sh Sentinel config

Slide 87

Slide 87 text

$> redis-‐cli -‐p 26379 127.0.0.1:26379> sentinel master mymaster 1) "name" 2) "mymaster" 3) "ip" 4) "127.0.0.1" 5) "port" 6) "6379" 7) "runid" 8) "e3a3fbab9687857c74266b360a85317aea2a213c" 9) "flags" 10) "master" 11) "pending-‐commands" 12) "0" 13) "last-‐ping-‐sent" 14) "0" 15) "last-‐ok-‐ping-‐reply" 16) "268" 17) "last-‐ping-‐reply" Discover master

Slide 88

Slide 88 text

$> redis-‐cli -‐p 26379 127.0.0.1:26379> sentinel slave mymaster 1) 1) "name" 2) "127.0.0.1:6380" 3) "ip" 4) "127.0.0.1" 5) "port" 6) "6380" 7) "runid" 8) "40a20d214c30b5a63a78882ee68d51f2b8506cc0" 9) "flags" 10) "slave" 11) "pending-‐commands" 12) "0" 13) "last-‐ping-‐sent" 14) "0" 15) "last-‐ok-‐ping-‐reply" 16) "158" 17) "last-‐ping-‐reply" 18) "158" Discover slaves Supports multiple slaves

Slide 89

Slide 89 text

$> redis-‐cli -‐p 26379 127.0.0.1:26379> SENTINEL get-‐master-‐addr-‐by-‐name mymaster 1) "127.0.0.1" 2) "6379" Get master host & port

Slide 90

Slide 90 text

Keepalived

Slide 91

Slide 91 text

Keepalived ✓Active-passive setups ✓Manages the virtual IP ✓Master gets virtual IP ✓Does failover ✓Uses check scripts

Slide 92

Slide 92 text

Keepalived vrrp_script chk_redis_master { script "/vagrant/keepalived.sh" interval 2 } vrrp_instance redis { state MASTER interface eth1 virtual_router_id 51 priority 150 advert_int 1 authentication { auth_type PASS auth_pass thepasswordispassword } virtual_ipaddress { 10.10.10.42 } track_script { chk_redis_master } } vrrp_script chk_redis_master { script "/vagrant/keepalived.sh" interval 2 } vrrp_instance redis { state MASTER interface eth1 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass thepasswordispassword } virtual_ipaddress { 10.10.10.42 } track_script { chk_redis_master } } Master Slave

Slide 93

Slide 93 text

Keepalived #!/bin/bash result=$(redis-‐cli -‐p 26379 SENTINEL get-‐master-‐addr-‐by-‐name mymaster | head -‐n1 2>&1) ip=`/sbin/ifconfig eth1 | grep 'inet addr:' | cut -‐d: -‐f2 | awk '{ print $1}'` if [ "$result" == "$ip" ] ; then echo "I'm master" exit 0 else echo "I'm slave" exit 1 fi check script

Slide 94

Slide 94 text

Cm_Cache_Backend_redis patch

Slide 95

Slide 95 text

https://github.com/ colinmollenhour/credis https://github.com/ThijsFeryn/ Cm_Cache_Backend_Redis/tree/ multiserver Multi-‐server support Library contains my patches Work in progress Sentinel support

Slide 96

Slide 96 text

Replacing Credis with Predis inside Cm_Cache_Backend_redis Would make more sense

Slide 97

Slide 97 text

Requires merging @vjom’s https:// github.com/ vmattila/predis/ tree/feature- sentinel feature branch

Slide 98

Slide 98 text

Twemproxy & Twemproxy agent https://github.com/Stono/redis-twemproxy-agent https://github.com/twitter/twemproxy

Slide 99

Slide 99 text

Twemproxy ✓Redis & Memcached Proxy ✓Written by Twitter ✓Does sharding ✓Persistent connections ✓Disable nodes on failure https://github.com/twitter/twemproxy

Slide 100

Slide 100 text

Twemproxy twem1: listen: 127.0.0.1:22122 hash: fnv1a_64 hash_tag: "{}" distribution: ketama auto_eject_hosts: false timeout: 400 redis: true servers: -‐ 10.10.10.40:6379:1 master1 -‐ 10.10.10.43:6379:1 master2

Slide 101

Slide 101 text

Twemproxy agent ✓NodeJS daemon ✓Monitors master status via Sentinel ✓Changes Twemproxy config on failover https://github.com/Stono/redis-twemproxy-agent

Slide 102

Slide 102 text

Twemproxy agent node cli.js -‐h 10.10.10.40 -‐p 26379 \ -‐f /etc/twemproxy/conf/magento.conf \ -‐c /etc/init.d/twemproxy restart \ -‐l /var/log/twemproxy/twemproxy_sentinel.log

Slide 103

Slide 103 text

Redis Cluster

Slide 104

Slide 104 text

Redis Cluster ✓In v3 ✓Sharding ✓Replication ✓Autofailover ✓Auto shard redirection ✓No need for Sentinel

Slide 105

Slide 105 text

port 7000 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes port 7001 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes port 7002 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes port 7003 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes port 7004 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes port 7005 cluster-‐enabled yes cluster-‐config-‐file nodes.conf cluster-‐node-‐timeout 5000 appendonly yes Cluster node config

Slide 106

Slide 106 text

$> ./redis-‐trib.rb create -‐-‐replicas 1 127.0.0.1:7000 \ 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 \ 127.0.0.1:7005 >>> Creating cluster Connecting to node 127.0.0.1:7000: OK Connecting to node 127.0.0.1:7001: OK Connecting to node 127.0.0.1:7002: OK Connecting to node 127.0.0.1:7003: OK Connecting to node 127.0.0.1:7004: OK Connecting to node 127.0.0.1:7005: OK >>> Performing hash slots allocation on 6 nodes... Using 3 masters: 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 Creating the cluster

Slide 107

Slide 107 text

$> redis-‐cli -‐c -‐p 7000 127.0.0.1:7000> set key value -‐> Redirected to slot [12539] located at 127.0.0.1:7002 OK 127.0.0.1:7000> get key "value" 127.0.0.1:7000> Using the cluster $> redis-‐cli -‐c -‐p 7005 127.0.0.1:7005> get key -‐> Redirected to slot [12539] located at 127.0.0.1:7002 "value" 127.0.0.1:7005> get key "value"