Slide 1
Slide 1 text
Making Magento go fast
By Thijs Feryn
Slide 2
Slide 2 text
my name
Thijs
Hi,
is
Slide 3
Slide 3 text
And I’m
@ThijsFeryn
on Twitter
Slide 4
Slide 4 text
I’m an
at
Evangelist
Slide 5
Slide 5 text
I’m
at
a board member
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
Magento is kinda slow …
Slide 8
Slide 8 text
But tremendously flexible
Slide 9
Slide 9 text
The first
attempts
Slide 10
Slide 10 text
First attempts
✓Install powerful machines
✓Lots of RAM & CPU
✓Loadbalancing on the webservers
✓Master/master replication on the
MySQL servers
✓High memory limit
✓APC byte code caching
Slide 11
Slide 11 text
Looking for
Magento
knowledge
Slide 12
Slide 12 text
First Magento
skills
✓Activate caching in the admin
panel
✓Flat catalogs
✓The Magento compiler
✓JS & CSS minification
Slide 13
Slide 13 text
The Magento
compiler is a lie!
Slide 14
Slide 14 text
Use a real byte code
cache instead
Slide 15
Slide 15 text
Things started to speed up
But it wasn’t fantastic
Slide 16
Slide 16 text
First conclusions
Slide 17
Slide 17 text
First conclusions
✓It uses a lot of RAM & CPU
✓And a whole lot of I/O
✓The MySQL server gets hammered
Slide 18
Slide 18 text
We
learned
how
to
cache
Slide 19
Slide 19 text
By default on disk in
in /var/cache
Slide 20
Slide 20 text
./mage-‐-‐c
./mage-‐-‐a
./mage-‐-‐4
./mage-‐-‐7
./mage-‐-‐e
./mage-‐-‐5
./mage-‐-‐b
./mage-‐-‐9
./mage-‐-‐0
./mage-‐-‐f
./mage-‐-‐3
./mage-‐-‐6
./mage-‐-‐8
./mage-‐-‐d
./mage-‐-‐1
Slide 21
Slide 21 text
./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_EN_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_NL_NL_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_EN_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_FR_EN_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_LU_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_STORE_ADMIN_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_STORE_BE_FR_CONFIG_CACHE
./mage-‐-‐8/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_sales_flat_shipment_grid_3
./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D
./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C
./mage-‐-‐d/mage-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1
./mage-‐-‐d/mage-‐-‐-‐MAGE_APP_E4D52B98688947405EDE639E947EE03D
./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_APP_B1FB6E8F13287C01E5C05063633DDA4C
./mage-‐-‐d/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_DB_PDO_MYSQL_DDL_log_url_info_1
./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_
./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_
./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_
./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencysymbol_
./mage-‐-‐1/mage-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_currencynumber_
./mage-‐-‐1/mage-‐-‐-‐internal-‐metadatas-‐-‐-‐MAGE_Zend_LocaleC_nl_NL_nametocurrency_
Slide 22
Slide 22 text
APC
Slide 23
Slide 23 text
In app/etc/local.xml
apc
MAGE_
Slide 24
Slide 24 text
Memcached
Slide 25
Slide 25 text
In app/etc/local.xml
Multiple servers
Slide 26
Slide 26 text
Still there are files in
/var/cache
Slide 27
Slide 27 text
2-level cache
✓Fast backend first (adapter)
✓Then the slow backend (disk)
Slide 28
Slide 28 text
/var/cache in tmpfs
mount
-‐t
tmpfs
-‐o
size=20m
tmpfs
/var/www/var/cache
Slide 29
Slide 29 text
is slow
Slide 30
Slide 30 text
Especially the older
versions
Slide 31
Slide 31 text
Magento doesn’t
officially work on the
latest versions
Slide 32
Slide 32 text
But we made it work
on 5.4. (with a patch)
http://www.magentocommerce.com/
knowledge-base/entry/php54-patch
Slide 33
Slide 33 text
Without byte code
caching, you’re in
trouble
Slide 34
Slide 34 text
We forgot that APC
doesn’t work on FastCGI
Slide 35
Slide 35 text
19280
33686
1.3
0.4
322052
110408
?
S
14:07
0:01
/usr/local/php-‐5.4/bin/php-‐cgi
18834
33696
0.0
0.0
267780
13608
?
S
14:07
0:00
/usr/local/php-‐5.4/bin/php-‐cgi
19280
33698
0.9
0.4
321808
110948
?
S
14:07
0:01
/usr/local/php-‐5.4/bin/php-‐cgi
19057
33700
0.0
0.0
268124
13416
?
S
14:07
0:00
/usr/local/php-‐5.4/bin/php-‐cgi
18767
33710
0.0
0.0
269728
14156
?
S
14:08
0:00
/usr/local/php-‐5.4/bin/php-‐cgi
Separate processes, no
joint cache
Slide 36
Slide 36 text
And we forgot that APC is
slowly dying
Slide 37
Slide 37 text
Seriously!
Slide 38
Slide 38 text
-FPM
Slide 39
Slide 39 text
root
26067
0.0
0.0
342892
6248
?
S
13:43
0:00
php-‐fpm:
master
process
(/etc/
php5/fpm/php-‐fpm.conf)
www-‐data
26068
0.2
1.1
396952
94684
?
S
13:43
0:03
php-‐fpm:
pool
www
www-‐data
26069
0.0
0.6
363472
56456
?
S
13:43
0:01
php-‐fpm:
pool
www
www-‐data
26087
0.0
0.6
362708
50188
?
S
13:44
0:00
php-‐fpm:
pool
www
Master proces that stores
the byte code cache
Slide 40
Slide 40 text
Opcache is awesome …
and a lot faster
Slide 41
Slide 41 text
Everything in memory!!!
Slide 42
Slide 42 text
We looked at HHVM
Slide 43
Slide 43 text
But database tuning is
also important
Slide 44
Slide 44 text
Which one will we choose?
Slide 45
Slide 45 text
1
pdo_mysql
mysql4
SET
NAMES
utf8
1
R/W
splitting
Slide 46
Slide 46 text
And all of a sudden,
we discovered Redis!
Slide 47
Slide 47 text
is GREAT!
✓Built-in replication
✓Data types (hash, list, …)
✓Save on disk
✓Session clustering
✓Cm_Cache_Backend_Redis by default
in Magento 1.8 CE & 1.13 EE
✓No 2-level cache
✓Multiple databases
✓Authentication
Slide 48
Slide 48 text
Cm_Cache_Backend_Redis
https://github.com/colinmollenhour/
Cm_Cache_Backend_Redis
Slide 49
Slide 49 text
In app/etc/local.xml
Cm_Cache_Backend_Redis
127.0.0.1
6379
1
0
1
10
0
1
1
20480
gzip
Slide 50
Slide 50 text
Cm_RedisSession
https://github.com/colinmollenhour/Cm_RedisSession
Slide 51
Slide 51 text
In app/etc/modules/Cm_RedisSession.xml
true
community
Slide 52
Slide 52 text
In app/etc/local.xml
db
127.0.0.1
6379
2.5
1
2048
gzip
1
6
5
30
7200
Slide 53
Slide 53 text
Varnish
Slide 54
Slide 54 text
Varnish
✓Only works on product catalog &
CMS pages
✓Doesn’t work on checkout
✓Great for static content
✓No SSL support
✓Best way to cache
Slide 55
Slide 55 text
Hit rate kinda sucked
Slide 56
Slide 56 text
Turpentine
https://github.com/nexcess/magento-turpentine
Slide 57
Slide 57 text
Turpentine
✓Community module for Varnish in
Magento
✓ESI or AJAX for session data
✓Configurable in the admin
✓VCL is pushed via Turpentine to the
Varnish over Telnet
✓Takes care of purging
✓SSL termination via Nginx
Slide 58
Slide 58 text
Lesti_FPC
When Varnish is not an option
https://github.com/GordonLesti/Lesti_Fpc
Slide 59
Slide 59 text
In app/etc/local.xml
86400
false
Cm_Cache_Backend_Redis
127.0.0.1
6379
cache-‐fpc
1
1
1
86400
10
1
1
gzip
Also
works
with
other
backends
Slide 60
Slide 60 text
Search
Slide 61
Slide 61 text
Full-text search on the
database is stupid …
and slow …
Slide 62
Slide 62 text
You could use
Slide 63
Slide 63 text
But I prefer
https://github.com/jreinke/magento-Elasticsearch
Slide 64
Slide 64 text
Host static files
separately (~CDN)
http://www.magentocommerce.com/magento-
connect/onepica-imagecdn-1.html
Slide 65
Slide 65 text
Horizontal scalability
Slide 66
Slide 66 text
Loadbalanced
ElasticSearch
Redis
Loadbalanced
MySQL MySQL
Loadbalanced
frontend
Nginx Nginx
Loadbalanced
Varnish Varnish
Loadbalanced
backend
Nginx Nginx
Client Admin
PHP-FPM PHP-FPM PHP-FPM PHP-FPM
SSL endpoint
ElasticSearch
Slide 67
Slide 67 text
Problem!
Slide 68
Slide 68 text
Cm_Cache_Backend_R
edis can’t handle
multiple instances
Slide 69
Slide 69 text
Got that
covered!
Slide 70
Slide 70 text
https://github.com/
colinmollenhour/credis
https://github.com/ThijsFeryn/
Cm_Cache_Backend_Redis/tree/
multiserver
Multi-‐
server
support
Library
contains
my
patches
Work
in
progress
Slide 71
Slide 71 text
Redis multi-server
✓Client-side hash
distribution
✓Master/slave
✓Sentinel
Slide 72
Slide 72 text
Sentinel
Slide 73
Slide 73 text
Sentinel
✓Discovery service for Redis servers
✓Part of the Redis core
✓Monitoring, notification & autofailover
✓Knows which are masters & slaves
✓Avoid problems when the master is
down
✓Avoids problems when to master is up
again
✓First connect to Sentinel, then to Redis
Slide 74
Slide 74 text
Loadbalanced
Redis
master
Client
Loadbalanced
Sentinel Sentinel
Redis
slave
Redis
slave
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
Magento v2?
Slide 77
Slide 77 text
My Redis multi-server
patch in production?
Slide 78
Slide 78 text
Redis v3 with built-in
clustering?
Slide 79
Slide 79 text
HHVM as the default
PHP runtime?
Slide 80
Slide 80 text
MariaDB as the default
MySQL server?
Slide 81
Slide 81 text
Varnish v4 support?
Slide 82
Slide 82 text
Use Consul for service
discovery?
Slide 83
Slide 83 text
And that
is how me
make
sure you
sleep at
night
Slide 84
Slide 84 text
But don’t ask us to
write Magento code
for you
Slide 85
Slide 85 text
We have partners for
that
Slide 86
Slide 86 text
But we can make your
Magento go fast
Slide 87
Slide 87 text
No content