Making Magento go fast

Making Magento go fast By Thijs Feryn

my name Thijs Hi, is

And I’m @ThijsFeryn on Twitter

I’m an at Evangelist

I’m at a board member

Magento is kinda slow …

But tremendously flexible

The first attempts

First attempts ✓Install powerful machines ✓Lots of RAM & CPU
✓Loadbalancing on the webservers ✓Master/master replication on the MySQL servers ✓High memory limit ✓APC byte code caching

Looking for Magento knowledge

First Magento skills ✓Activate caching in the admin panel ✓Flat
catalogs ✓The Magento compiler ✓JS & CSS minification

The Magento compiler is a lie!

Use a real byte code cache instead

Things started to speed up But it wasn’t fantastic

First conclusions

First conclusions ✓It uses a lot of RAM & CPU
✓And a whole lot of I/O ✓The MySQL server gets hammered

We learned how to cache

By default on disk in in /var/cache

./mage-‐-‐c ./mage-‐-‐a ./mage-‐-‐4 ./mage-‐-‐7 ./mage-‐-‐e
./mage-‐-‐5 ./mage-‐-‐b ./mage-‐-‐9 ./mage-‐-‐0 ./mage-‐-‐f ./mage-‐-‐3 ./mage-‐-‐6 ./mage-‐-‐8 ./mage-‐-‐d ./mage-‐-‐1

./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_NL_NL_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_FR_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_ADMIN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_sales_flat_shipment_grid_3 ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_

In app/etc/local.xml <global> <cache>
<backend>apc</backend> <prefix>MAGE_</prefix> </cache> </global>

Memcached

<global> <cache> <memcached>
<servers> <server> <host><![CDATA[127.0.0.1]]></host> <port><![CDATA[11211]]></port> <persistent><![CDATA[1]]></persistent> </server> </servers> </memcached> </cache> </global> In app/etc/local.xml Multiple servers

Still there are files in /var/cache

2-level cache ✓Fast backend first (adapter) ✓Then the slow backend
(disk)

/var/cache in tmpfs mount -‐t tmpfs -‐o size=20m tmpfs /var/www/var/cache

is slow

Especially the older versions

Magento doesn’t officially work on the latest versions

But we made it work on 5.4. (with a patch)
http://www.magentocommerce.com/ knowledge-base/entry/php54-patch

Without byte code caching, you’re in trouble

We forgot that APC doesn’t work on FastCGI

19280 33686 1.3 0.4 322052 110408 ? S 14:07
0:01 /usr/local/php-‐5.4/bin/php-‐cgi 18834 33696 0.0 0.0 267780 13608 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 19280 33698 0.9 0.4 321808 110948 ? S 14:07 0:01 /usr/local/php-‐5.4/bin/php-‐cgi 19057 33700 0.0 0.0 268124 13416 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 18767 33710 0.0 0.0 269728 14156 ? S 14:08 0:00 /usr/local/php-‐5.4/bin/php-‐cgi Separate processes, no joint cache

And we forgot that APC is slowly dying

Seriously!

root 26067 0.0 0.0 342892 6248 ? S 13:43 0:00
php-‐fpm: master process (/etc/ php5/fpm/php-‐fpm.conf) www-‐data 26068 0.2 1.1 396952 94684 ? S 13:43 0:03 php-‐fpm: pool www www-‐data 26069 0.0 0.6 363472 56456 ? S 13:43 0:01 php-‐fpm: pool www www-‐data 26087 0.0 0.6 362708 50188 ? S 13:44 0:00 php-‐fpm: pool www Master proces that stores the byte code cache

Opcache is awesome … and a lot faster

Everything in memory!!!

We looked at HHVM

But database tuning is also important

Which one will we choose?

<global> <resources> <default_setup>
<connection> <host><![CDATA[Master-‐host]]></host> <username><![CDATA[user]]></username> <password><![CDATA[pass]]></password> <dbname><![CDATA[magentodb]]></dbname> <active>1</active> </connection> </default_setup> <default_read> <connection> <use/> <host><![CDATA[Slave-‐host]]></host> <username><![CDATA[user]]></username> <password><![CDATA[pass]]></password> <dbname><![CDATA[magento]]></dbname> <type>pdo_mysql</type> <model>mysql4</model> <initStatements>SET NAMES utf8</initStatements> <active>1</active> </connection> </default_read> </resources> </global> R/W splitting

And all of a sudden, we discovered Redis!

is GREAT! ✓Built-in replication ✓Data types (hash, list, …) ✓Save
on disk ✓Session clustering ✓Cm_Cache_Backend_Redis by default in Magento 1.8 CE & 1.13 EE ✓No 2-level cache ✓Multiple databases ✓Authentication

Cm_Cache_Backend_Redis https://github.com/colinmollenhour/ Cm_Cache_Backend_Redis

In app/etc/local.xml <global> <cache>
<backend>Cm_Cache_Backend_Redis</backend> <backend_options> <server>127.0.0.1</server> <port>6379</port> <persistent></persistent> <database>1</database> <password></password> <force_standalone>0</force_standalone> <connect_retries>1</connect_retries> <read_timeout>10</read_timeout> <automatic_cleaning_factor>0</automatic_cleaning_factor> <compress_data>1</compress_data> <compress_tags>1</compress_tags> <compress_threshold>20480</compress_threshold> <compression_lib>gzip</compression_lib> </backend_options> </cache> </global>

Cm_RedisSession https://github.com/colinmollenhour/Cm_RedisSession

In app/etc/modules/Cm_RedisSession.xml <config> <modules>
<Cm_RedisSession> <active>true</active> <codePool>community</codePool> </Cm_RedisSession> </modules> </config>

In app/etc/local.xml <global> <session_save>db</session_save> <redis_session>
<host>127.0.0.1</host> <port>6379</port> <password></password> <timeout>2.5</timeout> <persistent></persistent> <db>1</db> <compression_threshold>2048</compression_threshold> <compression_lib>gzip</compression_lib> <log_level>1</log_level> <max_concurrency>6</max_concurrency> <break_after_frontend>5</break_after_frontend> <break_after_adminhtml>30</break_after_adminhtml> <bot_lifetime>7200</bot_lifetime> </redis_session> </global>

Varnish

Varnish ✓Only works on product catalog & CMS pages ✓Doesn’t
work on checkout ✓Great for static content ✓No SSL support ✓Best way to cache

Hit rate kinda sucked

Turpentine https://github.com/nexcess/magento-turpentine

Turpentine ✓Community module for Varnish in Magento ✓ESI or AJAX
for session data ✓Configurable in the admin ✓VCL is pushed via Turpentine to the Varnish over Telnet ✓Takes care of purging ✓SSL termination via Nginx

Lesti_FPC When Varnish is not an option https://github.com/GordonLesti/Lesti_Fpc

In app/etc/local.xml <global> <fpc>
<lifetime>86400</lifetime> <debug>false</debug> <backend>Cm_Cache_Backend_Redis</backend> <backend_options> <server>127.0.0.1</server> <port>6379</port> <persistent>cache-‐fpc</persistent> <database>1</database> <password></password> <force_standalone>1</force_standalone> <connect_retries>1</connect_retries> <lifetimelimit>86400</lifetimelimit> <read_timeout>10</read_timeout> <compress_data>1</compress_data> <compress_tags>1</compress_tags> <compress_data>gzip</compress_data> </backend_options> </fpc> </global> Also works with other backends

Search

Full-text search on the database is stupid … and slow
…

You could use

But I prefer https://github.com/jreinke/magento-Elasticsearch

Host static files separately (~CDN) http://www.magentocommerce.com/magento- connect/onepica-imagecdn-1.html

Horizontal scalability

Loadbalanced ElasticSearch Redis Loadbalanced MySQL MySQL Loadbalanced frontend Nginx Nginx
Loadbalanced Varnish Varnish Loadbalanced backend Nginx Nginx Client Admin PHP-FPM PHP-FPM PHP-FPM PHP-FPM SSL endpoint ElasticSearch

Problem!

Cm_Cache_Backend_R edis can’t handle multiple instances

Got that covered!

https://github.com/ colinmollenhour/credis https://github.com/ThijsFeryn/ Cm_Cache_Backend_Redis/tree/ multiserver Multi-‐ server support Library
contains my patches Work in progress

Redis multi-server ✓Client-side hash distribution ✓Master/slave ✓Sentinel

Sentinel

Sentinel ✓Discovery service for Redis servers ✓Part of the Redis
core ✓Monitoring, notification & autofailover ✓Knows which are masters & slaves ✓Avoid problems when the master is down ✓Avoids problems when to master is up again ✓First connect to Sentinel, then to Redis

Loadbalanced Redis master Client Loadbalanced Sentinel Sentinel Redis slave Redis
slave

Magento v2?

My Redis multi-server patch in production?

Redis v3 with built-in clustering?

HHVM as the default PHP runtime?

MariaDB as the default MySQL server?

Varnish v4 support?

Use Consul for service discovery?

And that is how me make sure you sleep at
night

But don’t ask us to write Magento code for you

We have partners for that

But we can make your Magento go fast

Making Magento go fast

Making Magento go fast

More Decks by Thijs Feryn

Other Decks in Technology

Featured

Transcript