Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Making Magento go fast
Search
Thijs Feryn
June 05, 2014
Technology
2
440
Making Magento go fast
Slides for my Magento performance talk at Belgento 2014
Thijs Feryn
June 05, 2014
Tweet
Share
More Decks by Thijs Feryn
See All by Thijs Feryn
Caching the uncacheable with Varnish - PHP London 2020
thijsferyn
0
450
Accelerating OTT video platforms with Varnish - London Video Tech meetup 2020
thijsferyn
0
390
't Oncachebare cachen
thijsferyn
0
380
Caching the uncacheable with Varnish - PHP UG FFM 19
thijsferyn
1
760
Developing cacheable PHP applications - PHP Barcelona 2019
thijsferyn
0
670
Caching the uncacheable with Varnish - FullstackEU 2019
thijsferyn
0
480
Varnish beyond basic web acceleration - Symfony Live Berlin 2019
thijsferyn
0
400
Developing cacheable PHP applications
thijsferyn
0
450
Varnish beyond basic web acceleration - DAHO.AM 2019
thijsferyn
0
420
Other Decks in Technology
See All in Technology
NLPコロキウム20251022_超効率化への挑戦: LLM 1bit量子化のロードマップ
yumaichikawa
1
130
サイバーエージェント流クラウドコスト削減施策「みんなで金塊堀太郎」
kurochan
4
2.2k
ソースを読むプロセスの例
sat
PRO
15
9.6k
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
12
81k
難しいセキュリティ用語をわかりやすくしてみた
yuta3110
0
340
AIエージェント入門 〜基礎からMCP・A2Aまで〜
shukob
0
120
FinOps について (ちょっと) 本気出して考えてみた
skmkzyk
0
180
あなたの知らない Linuxカーネル脆弱性の世界
recruitengineers
PRO
3
120
ローカルLLMとLINE Botの組み合わせ その2(EVO-X2でgpt-oss-120bを利用) / LINE DC Generative AI Meetup #7
you
PRO
0
130
ビズリーチ求職者検索におけるPLMとLLMの活用 / Search Engineering MEET UP_2-1
visional_engineering_and_design
1
180
AI時代におけるデータの重要性 ~データマネジメントの第一歩~
ryoichi_ota
0
700
Codexとも仲良く。CodeRabbit CLIの紹介
moongift
PRO
1
260
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
The Straight Up "How To Draw Better" Workshop
denniskardys
238
140k
Product Roadmaps are Hard
iamctodd
PRO
55
11k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
The Cult of Friendly URLs
andyhume
79
6.6k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
Building an army of robots
kneath
306
46k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Why Our Code Smells
bkeepers
PRO
340
57k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
140
34k
Keith and Marios Guide to Fast Websites
keithpitt
411
23k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Transcript
Making Magento go fast By Thijs Feryn
my name Thijs Hi, is
And I’m @ThijsFeryn on Twitter
I’m an at Evangelist
I’m at a board member
None
Magento is kinda slow …
But tremendously flexible
The first attempts
First attempts ✓Install powerful machines ✓Lots of RAM & CPU
✓Loadbalancing on the webservers ✓Master/master replication on the MySQL servers ✓High memory limit ✓APC byte code caching
Looking for Magento knowledge
First Magento skills ✓Activate caching in the admin panel ✓Flat
catalogs ✓The Magento compiler ✓JS & CSS minification
The Magento compiler is a lie!
Use a real byte code cache instead
Things started to speed up But it wasn’t fantastic
First conclusions
First conclusions ✓It uses a lot of RAM & CPU
✓And a whole lot of I/O ✓The MySQL server gets hammered
We learned how to cache
By default on disk in in /var/cache
./mage-‐-‐c ./mage-‐-‐a ./mage-‐-‐4 ./mage-‐-‐7 ./mage-‐-‐e
./mage-‐-‐5 ./mage-‐-‐b ./mage-‐-‐9 ./mage-‐-‐0 ./mage-‐-‐f ./mage-‐-‐3 ./mage-‐-‐6 ./mage-‐-‐8 ./mage-‐-‐d ./mage-‐-‐1
./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_NL_NL_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_FR_EN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_ADMIN_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE ./mage-‐-‐8/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_sales_flat_shipment_grid_3 ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C ./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1 ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_ ./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_ ./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_
APC
In app/etc/local.xml <global> <cache>
<backend>apc</backend> <prefix>MAGE_</prefix> </cache> </global>
Memcached
<global> <cache> <memcached>
<servers> <server> <host><![CDATA[127.0.0.1]]></host> <port><![CDATA[11211]]></port> <persistent><![CDATA[1]]></persistent> </server> </servers> </memcached> </cache> </global> In app/etc/local.xml Multiple servers
Still there are files in /var/cache
2-level cache ✓Fast backend first (adapter) ✓Then the slow backend
(disk)
/var/cache in tmpfs mount -‐t tmpfs -‐o size=20m tmpfs /var/www/var/cache
is slow
Especially the older versions
Magento doesn’t officially work on the latest versions
But we made it work on 5.4. (with a patch)
http://www.magentocommerce.com/ knowledge-base/entry/php54-patch
Without byte code caching, you’re in trouble
We forgot that APC doesn’t work on FastCGI
19280 33686 1.3 0.4 322052 110408 ? S 14:07
0:01 /usr/local/php-‐5.4/bin/php-‐cgi 18834 33696 0.0 0.0 267780 13608 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 19280 33698 0.9 0.4 321808 110948 ? S 14:07 0:01 /usr/local/php-‐5.4/bin/php-‐cgi 19057 33700 0.0 0.0 268124 13416 ? S 14:07 0:00 /usr/local/php-‐5.4/bin/php-‐cgi 18767 33710 0.0 0.0 269728 14156 ? S 14:08 0:00 /usr/local/php-‐5.4/bin/php-‐cgi Separate processes, no joint cache
And we forgot that APC is slowly dying
Seriously!
-FPM
root 26067 0.0 0.0 342892 6248 ? S 13:43 0:00
php-‐fpm: master process (/etc/ php5/fpm/php-‐fpm.conf) www-‐data 26068 0.2 1.1 396952 94684 ? S 13:43 0:03 php-‐fpm: pool www www-‐data 26069 0.0 0.6 363472 56456 ? S 13:43 0:01 php-‐fpm: pool www www-‐data 26087 0.0 0.6 362708 50188 ? S 13:44 0:00 php-‐fpm: pool www Master proces that stores the byte code cache
Opcache is awesome … and a lot faster
Everything in memory!!!
We looked at HHVM
But database tuning is also important
Which one will we choose?
<global> <resources> <default_setup>
<connection> <host><![CDATA[Master-‐host]]></host> <username><![CDATA[user]]></username> <password><![CDATA[pass]]></password> <dbname><![CDATA[magentodb]]></dbname> <active>1</active> </connection> </default_setup> <default_read> <connection> <use/> <host><![CDATA[Slave-‐host]]></host> <username><![CDATA[user]]></username> <password><![CDATA[pass]]></password> <dbname><![CDATA[magento]]></dbname> <type>pdo_mysql</type> <model>mysql4</model> <initStatements>SET NAMES utf8</initStatements> <active>1</active> </connection> </default_read> </resources> </global> R/W splitting
And all of a sudden, we discovered Redis!
is GREAT! ✓Built-in replication ✓Data types (hash, list, …) ✓Save
on disk ✓Session clustering ✓Cm_Cache_Backend_Redis by default in Magento 1.8 CE & 1.13 EE ✓No 2-level cache ✓Multiple databases ✓Authentication
Cm_Cache_Backend_Redis https://github.com/colinmollenhour/ Cm_Cache_Backend_Redis
In app/etc/local.xml <global> <cache>
<backend>Cm_Cache_Backend_Redis</backend> <backend_options> <server>127.0.0.1</server> <port>6379</port> <persistent></persistent> <database>1</database> <password></password> <force_standalone>0</force_standalone> <connect_retries>1</connect_retries> <read_timeout>10</read_timeout> <automatic_cleaning_factor>0</automatic_cleaning_factor> <compress_data>1</compress_data> <compress_tags>1</compress_tags> <compress_threshold>20480</compress_threshold> <compression_lib>gzip</compression_lib> </backend_options> </cache> </global>
Cm_RedisSession https://github.com/colinmollenhour/Cm_RedisSession
In app/etc/modules/Cm_RedisSession.xml <config> <modules>
<Cm_RedisSession> <active>true</active> <codePool>community</codePool> </Cm_RedisSession> </modules> </config>
In app/etc/local.xml <global> <session_save>db</session_save> <redis_session>
<host>127.0.0.1</host> <port>6379</port> <password></password> <timeout>2.5</timeout> <persistent></persistent> <db>1</db> <compression_threshold>2048</compression_threshold> <compression_lib>gzip</compression_lib> <log_level>1</log_level> <max_concurrency>6</max_concurrency> <break_after_frontend>5</break_after_frontend> <break_after_adminhtml>30</break_after_adminhtml> <bot_lifetime>7200</bot_lifetime> </redis_session> </global>
Varnish
Varnish ✓Only works on product catalog & CMS pages ✓Doesn’t
work on checkout ✓Great for static content ✓No SSL support ✓Best way to cache
Hit rate kinda sucked
Turpentine https://github.com/nexcess/magento-turpentine
Turpentine ✓Community module for Varnish in Magento ✓ESI or AJAX
for session data ✓Configurable in the admin ✓VCL is pushed via Turpentine to the Varnish over Telnet ✓Takes care of purging ✓SSL termination via Nginx
Lesti_FPC When Varnish is not an option https://github.com/GordonLesti/Lesti_Fpc
In app/etc/local.xml <global> <fpc>
<lifetime>86400</lifetime> <debug>false</debug> <backend>Cm_Cache_Backend_Redis</backend> <backend_options> <server>127.0.0.1</server> <port>6379</port> <persistent>cache-‐fpc</persistent> <database>1</database> <password></password> <force_standalone>1</force_standalone> <connect_retries>1</connect_retries> <lifetimelimit>86400</lifetimelimit> <read_timeout>10</read_timeout> <compress_data>1</compress_data> <compress_tags>1</compress_tags> <compress_data>gzip</compress_data> </backend_options> </fpc> </global> Also works with other backends
Search
Full-text search on the database is stupid … and slow
…
You could use
But I prefer https://github.com/jreinke/magento-Elasticsearch
Host static files separately (~CDN) http://www.magentocommerce.com/magento- connect/onepica-imagecdn-1.html
Horizontal scalability
Loadbalanced ElasticSearch Redis Loadbalanced MySQL MySQL Loadbalanced frontend Nginx Nginx
Loadbalanced Varnish Varnish Loadbalanced backend Nginx Nginx Client Admin PHP-FPM PHP-FPM PHP-FPM PHP-FPM SSL endpoint ElasticSearch
Problem!
Cm_Cache_Backend_R edis can’t handle multiple instances
Got that covered!
https://github.com/ colinmollenhour/credis https://github.com/ThijsFeryn/ Cm_Cache_Backend_Redis/tree/ multiserver Multi-‐ server support Library
contains my patches Work in progress
Redis multi-server ✓Client-side hash distribution ✓Master/slave ✓Sentinel
Sentinel
Sentinel ✓Discovery service for Redis servers ✓Part of the Redis
core ✓Monitoring, notification & autofailover ✓Knows which are masters & slaves ✓Avoid problems when the master is down ✓Avoids problems when to master is up again ✓First connect to Sentinel, then to Redis
Loadbalanced Redis master Client Loadbalanced Sentinel Sentinel Redis slave Redis
slave
None
Magento v2?
My Redis multi-server patch in production?
Redis v3 with built-in clustering?
HHVM as the default PHP runtime?
MariaDB as the default MySQL server?
Varnish v4 support?
Use Consul for service discovery?
And that is how me make sure you sleep at
night
But don’t ask us to write Magento code for you
We have partners for that
But we can make your Magento go fast
None