From Bottlenecks to Breakthroughs: How the New York Post Mastered Scalability

Presented at WordCamp US, Oregon, 2024

https://us.wordcamp.org/2024/session/from-bottlenecks-to-breakthroughs-how-the-new-york-post-mastered-scalability/

https://alley.com/

Transcript

1. From Bottlenecks to Breakthroughs: How the New York Post Mastered Scalability Presented by Linnea Huxford
2. Agenda The New York Post’s WordPress Journey An Overview of Scaling Strategies Deep Dive into Bottlenecks & Solutions
3. Setting the Stage: The New York Post’s WordPress Journey
4. A long history of building and publishing with WordPress Number of years on WordPress 11 Years Database rows for the WP_Posts table 15 Million
5. Timeline 2013 Launched The NY Post on WordPress 2015 Redesigned NY Post & Page Six 2020 Migrated sites from VIP Classic to VIP Go 2021 A/B tests and rollout of the 2021 NY Post redesign 2023 Launched the redesign of Page Six as a child theme
6. Massive content volume Post tags 28,000 Articles 1,299,949 Database Size 244 GB
7. Mega pageviews and sudden traffic surges from Breaking News
8. Overview of Scaling Strategies
9. We run a WordPress Multisite on WordPress VIP with a “monolith” architecture • Gutenberg block editor with custom blocks • WP VIPʼs mu-plugins have many performance tweaks to support enterprise usage • Persistent object cache - memcached • Reliable Cron System with Automatticʼs Cron Control plugin • CDN for images to deliver them quickly from servers close to the userʼs location • Robust Page Cache to serve up pre-built HTML to the majority of users.
10. We’ve succeeded at very effective use of the Persistent Object Cache, achieving 98% hit rate!
11. A typical page request has very few database queries because most data is already in the object cache.
12. Key Principle: No database writes from frontend traffic • API integrations often hook in on admin actions. For example, when posts are saved or updated, using the wp_after_insert_post hook. • Scheduled cron jobs update data from external sources. • To optimize performance, API data is stored in the object cache, minimizing the need for repeated external requests • Rewrite rules are updated via a CLI command • Analytics, comments, and membership functionalities are powered by external third parties, leveraging JavaScript HTTP calls and scripts instead of backend calls to WordPress.
13. We use the following Elasticsearch Plugins for better query performance SearchPress Indexes content in Elasticsearch ES WP Query Offloads query to Elasticsearch https://github.com/alleyinteractive/searchpress https://github.com/alleyinteractive/es-wp-query ES_WP_Query
14. Simply add ‘es’ => true and the query will use Elasticsearch to get the Post IDs Elasticsearch DSL Elasticsearch Cluster Post ID Post ID Post ID Post ID … Complex Query: Metro Snowstorm Posts with Video
15. ES WP Query will then shoehorn the Post IDs from Elasticsearch into a highly efficient post__in query to retrieve the full WP_Post objects from the database. es-wp-query/class-es-wp-query-shoehorn.php Post ID Post ID Post ID Post ID …
16. We built Remote Backstop as a safety net https://github.com/alleyinteractive/remote-backstop Object Cache Object Cache Object Cache Object Cache Object Cache Remote Services Remote Backstop
17. Remote backstop has several key benefits 1. Site Stability: The plugin prevents site downtime when external resources are unavailable. 2. Stress Reduction: Remote Backstop alleviates the load on a stressed resource by serving cached responses. 3. User Experience: End users are unaffected, unaware of any issues, due to the seamless fallback mechanism.
18. Deep Dive into Bottlenecks & Solutions
19. We use New Relic to monitor server performance
20. The WP New Relic Transactions plugin adds WordPress context to New Relic https://github.com/alleyinteractive/wp-new-relic-transactions
21. Identifying high-impact endpoints: A New Relic analysis
22. Bottlenecks & Quick Solutions
23. Bottleneck 1 50% of this request was checking for AMP
24. Bandaid Applied
25. Solution We used the PHP Static Keyword
26. Bottleneck 2 Simultaneous cache expiry The homepage is very long and each module’s query is cached. If all of the caches expire at the same time, the page becomes slow to build.
27. Solution Implementing staggered cache durations Random cache duration leads to more evenly distributed cache expiration, preventing server overload.
28. Bottleneck 3 Recursive function calls inside loops Solution Removing recursion
29. Bottleneck 4 Time to build 404 pages
30. Solution Cache the HTML for the 404 page https://github.com/alleyinteractive/wp-404-caching
31. Bottleneck 5 Slow database queries
32. Solutions to slow queries • Solution 1 Avoid posts__not_in ◦ Get more posts than needed and exclude posts in PHP. • Solution 2 Disable Getting All Found Rows ◦ If Pagination isnʼt necessary, set 'no_found_rows' => true • Solution 3 Optimize Query Parameters ◦ 'ignore_sticky_posts' => true ◦ 'suppress_filters' => false • Solution 4 Offload to Elasticsearch and Cache Results • Solution 5 Avoid Unnecessary Queries ◦ Whenever possible, avoid running the query entirely. Sometimes, plugins add unnecessary functionality that can be disabled to improve performance.
33. Set performant query arguments as defaults
34. Query Performance Comparison
35. Add a Date Query • Getting 100 Recent Posts without a date query: 171 ms • Limiting the query to the last 3 months: 46 ms
36. Conclusion WordPress, coupled with WordPress VIP hosting, offers a robust foundation for building enterprise solutions that scale effectively. Weʼve relied on several key principles to achieve this scalability: • Leveraging Elasticsearch to enhance WP Query performance and offload complex queries. • Utilizing Object Cache to efficiently store data from queries and external APIs, reducing load times. • Adhering to the principle of ‘No Frontend Database Writes,ʼ ensuring data integrity and performance. • Identifying bottlenecks using tools like New Relic and Query Monitor to pinpoint and resolve performance issues. • Applying common solutions such as query optimizations and caching strategies to further boost efficiency.
37. “Great things are not done by impulse, but by a series of small things brought together.ˮ — Vincent Van Gogh
38. Thank You! We’re hiring software developers! https://alley.com/careers/