Slide 1
Slide 1 text
Scaling Magento
Reid Parham, Aaron Edmonds, and Kyle Terry
Public distribution: sensitive information omitted.
Slide 2
Slide 2 text
COPIOUS
● User-Centered Digital Experience Agency
● Strategy
● Experience
● Engineering
● http://copio.us/about
Slide 3
Slide 3 text
Scale Your Code
A.K.A. Magento is hard
Slide 4
Slide 4 text
Code Management
● Magento is big!
○ Our project has over 820,000 lines of PHP
● Multi-lingual, multi-currency, multi-store
● Classes can have complex names
○ *cough*
Enterprise_Reward_Block_Adminhtml_Customer_Edit_T
ab_Reward_History_Grid_Column_Renderer_Reason
*cough*
Slide 5
Slide 5 text
Code Management (cont.)
● Configuration is driven by XML
● The dreaded EAV
● Magento Indices
● Event-Observer
Slide 6
Slide 6 text
Code Management (Tools)
Good tools make the job easier!
● A good IDE
○ Magicento
● Commerce Bug 2
● n98-magerun
Slide 7
Slide 7 text
Code Management
● NEVER modify core files
○ Magento’s forum never helped
● NEVER* add files to app/code/local/Mage
○ Magento was built to be modular**
● Test your code with flat catalog enabled
and disabled
● Before overwriting classes, check for events
Slide 8
Slide 8 text
Code Optimization (Quick Wins)
Caching Magento Blocks
● DIY! Event to add cache data:
core_block_abstract_to_html_before
● OR use a module https://github.
com/aligent/CacheObserver
Slide 9
Slide 9 text
Code Optimization (Quick Wins)
Mage::getModel(‘catalog/product’)->load($_product-
>getId());
● This is bad in templates and when looping
over product collections
● Load with initial data select
○ used_in_product_listing attribute option
Slide 10
Slide 10 text
Code Optimization
Make efficient use of Magento indices
● Example: Catalog URL Rewrites
○ Includes all products by default (including products
marked as “Not Visible Individually”)
○ Do you need SEO friendly URLs for products that
will never be seen???
○ Reduce your index size by up to 95%
○ Mage_Catalog_Model_Resource_Url::_getProducts
Slide 11
Slide 11 text
Code Optimization (Quick Wins)
Mage_Catalog_Model_Resource_Product_Typ
e_Configurable_Product_Collection::
isEnabledFlat?
FALSE
Slide 12
Slide 12 text
Systems
● Hardware Profile
● Cluster Design
● Scaling
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
Hardware Profile (overview)
● 2 racks of hardware and dozens of servers
● Top quality of available (and compatible)
chipsets and memory
● Buffered DDR3; 1 channel per CPU
● 126 kW of stable, reliable, redundant, and
backed up power
● Minor kernel tweaks
Slide 15
Slide 15 text
Hardware Profile (network)
● NetScaler for load balancing
○ Vserver pools
○ Balances web, database, admin and endeca
○ Monitors will remove downed hosts
● Redundant Network Infrastructure
○ Backplane uses LACP (link aggregation) for
redundancy, load balancing and failover
○ HA pairing of configurations
Slide 16
Slide 16 text
Hardware Profile (network)
Dynamic port forwarding for browsing:
kyle@localhost $ ssh -L 2221:127.0.0.1:2221 whitelistedhost.example.com
kyle@whitelistedhost $ ssh -D 2221 cluster.example.com
Static port forwarding for Navicat SSH tunneling (tunneling through a tunnel):
kyle@localhost $ ssh -L 2222:127.0.0.1:2222 whitelistedhost.example.com
kyle@whitelistedhost $ ssh -L 2222:127.0.0.1:22 cluster.example.com
Slide 17
Slide 17 text
Hardware Profile (web)
● Dual Intel Xeon E3-1230 @ 3.30GHz
● 32 GB RAM
● Dozens of servers
● nginx and PHP5-FPM
● 6:1 ratio of PHP processes to CPU cores
Slide 18
Slide 18 text
Hardware Profile (database)
● Redundant database hosts
● MySQL 5.6 chosen for scaling capability
● tcmalloc further improves throughput
● Master/slave replication
● Standby hosts for warm failover
● Failure point: > 4,000 checkouts/hour
Slide 19
Slide 19 text
Hardware Profile (database)
● Quad Intel Xeon E7-2860
○ 10 cores + hyperthreading each totalling 80 threads
● 128 GBs of RAM
● RAID10 SSDs for data
○ writeback cache; noatime,noexec mount options
● RAID1 HDDs for OS
Slide 20
Slide 20 text
Oops!
Slide 21
Slide 21 text
Hardware Profile (cache)
● Powering discrete instances of Redis
○ Sessions
○ Full page cache
○ Magento back end cache
○ Background processing queues
● Discrete instances are for threading, differing
memory limits, differing backup rules, and
multi-db deprecation
Slide 22
Slide 22 text
Hardware Profile (cache)
● Content is compressed with LZF
○ Compression and decompression with LZF is faster
than gzip so it’s an ideal solution
● Decreased utilization of network capacity
● Sentinel for failover (soon)
● RDB BGSAVE: prime number intervals
Slide 23
Slide 23 text
Compression Outcomes
Slide 24
Slide 24 text
Hardware Profile (cache)
● Quad Intel Xeon E5-2620 @ 2.00GHz
● 128 GBs of RAM
● 4 bonded network interfaces
○ Prevents saturation of private network
○ 4 Gb/s
○ Bonding mode 5 (balance-tlb)
■ No special switch support
■ Nice when the colo manages the switch
Slide 25
Slide 25 text
Hardware Profile (utility)
● Cron and systems jobs
● Scripts
● Deploys
● Chef Server 10 for deploy and configuration
● Tests
○ Database test suite in Perl (Test::DatabaseRow)
● Backups (and copies)
Slide 26
Slide 26 text
Cluster Overview
● Production
○ Most hardware serves production
● Staging
○ Some data promoted to production nightly
● Preview{1..n}
○ Instances for testing and previewing new features,
bug fixes and design changes.
Slide 27
Slide 27 text
● Aggregate hardware availability exceeds
six nines (99.9999%)
● Software availability is ~99.999%
● Software, including deployments: 99.98%
● Software, including maintenance: 99.9%
● Non-recoverable human errors: 98%
Production Uptime
Slide 28
Slide 28 text
Scale Your Team
Slide 29
Slide 29 text
Team Profile
● 16 committers; 8.25 FTE
● 4 Project Managers
● 5 departments
● 31 vendors
● 5 time zones
Slide 30
Slide 30 text
Team Values
● State your needs; respect others’
● Respect is given, then adjusted
● Process can always change and improve
● Work/life balance
● Mature and non-aggressive; mediate conflict
● Honesty and transparency
Slide 31
Slide 31 text
Team Mantras
● Trust (relevant) data; make things visible
● Measurable, repeatable, falsifiable
(scientific method)
● Redundancy reduces risks (if documented)
● Set expectations (timing, contents, formats)
and deliver on them
Slide 32
Slide 32 text
Team Mantras
● Automate what is repeated
● Use known patterns and
proven architectures
● Grow talent from within
● Compartmentalization of some data,
code, and knowledge
Slide 33
Slide 33 text
No content
Slide 34
Slide 34 text
10 Integrated Vendors
Adobe, Akamai, tax calculation,
legacy software, Ebay, gift cards,
ERP (fulfillment and inventory), Oracle,
Tierpoint (Dallas, Seattle, Spokane),
Endeca provider
Slide 35
Slide 35 text
advertising, application analytics, email,
hardware analysis and functionality, maps,
offsite storage, promotions, payment gateways,
remarketing, shipping estimates, SMS,
social networks, uptime
21 Accessory Vendors
Slide 36
Slide 36 text
● Group emails: avoid general questions,
assign actions to people, minimize
distribution lists
● Identify urgency of requests
● Use email filters
● Coach and mentor
Effective Communication
Slide 37
Slide 37 text
● Daily phone calls: only while needed
● Set an agenda; keep to a schedule
● Encourage people to skip calls
or to leave early
● End the call when completed
Effective Communication
Slide 38
Slide 38 text
Tools
● GitHub
● Google Docs
● Pivotal Tracker
● Conference calls, Skype, and IM
● BugHerd
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
Launch Day
Slide 42
Slide 42 text
Release Day
Slide 43
Slide 43 text
QA preparation
Slide 44
Slide 44 text
Productive!
Slide 45
Slide 45 text
Off-hours chaos
Slide 46
Slide 46 text
Build Knowledge
● Document the “obvious”
● 1000-line README
● Capture failures and solutions
● What happens when?
● Which database and server?
Slide 47
Slide 47 text
Automation Schedule
Slide 48
Slide 48 text
“This is how we work.”
Slide 49
Slide 49 text
Example Git Workflow
Slide 50
Slide 50 text
Learn from previous failures.
Slide 51
Slide 51 text
Code Review
● Standardize pull request structures
● Constructive feedback; ask questions
● emoji-cheat-sheet.com
Slide 52
Slide 52 text
Code Review
Pull requests can also be workspaces
Slide 53
Slide 53 text
Releases and Git flow: rhythm, ownership, and pride.
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
Deployments
● Monday through Thursday only!
● Communication: tickets, cross references,
pull requests, QA status, and releases
● Set expectations: timings for outages,
maintenance, and degraded functionality
● Are we done, yet?
● Explain outcomes and options
Slide 56
Slide 56 text
Community Participation
● Patches submitted
○ Redis
○ Cm_RedisSession
○ Cm_Cache_Backend_Redis
○ https://github.com/magento/magento2
● Modules improved
○ CacheObserver
○ VF_CustomMenu
Slide 57
Slide 57 text
Community Participation
● http://magento.stackexchange.com/
● http://stackoverflow.com/
● phpredis bug(s)
Slide 58
Slide 58 text
● Spence, Muneera U. Collaborative
Processes lecture. 13 Apr. 2006.
● Marks, Andrea. "The Role of Writing in a
Design Curriculum." AIGA: Design Education
(2004).
● Katzenbach, Jon R., and Douglas K. Smith.
The Wisdom of Teams. HarperCollins, 2003.
Collaboration Texts
Slide 59
Slide 59 text
● Bennis, Warren, and Patricia W. Biederman.
Organizing Genius. Perseus, 1997.
● Marcum, James W. After the Information
Age. Peter Lang, 2006.
● https://en.wikipedia.org/wiki/Collaboration
(and collaborative method)
Collaboration Texts
Slide 60
Slide 60 text
See Also
GitHub (and Gist)
@parhamr
@kyleterry
@aedmonds
Slide 61
Slide 61 text
Questions?