A Neos CMS performance detective story

I'm showing how I solved 3 very exciting performance issues in Neos CMS projects and this way made the work for the editors of the system so much easier.
Also every performance fix was added to the Neos core, so everyone using the Open Source CMS has a better system now. That's what Open Source is about!

1. LET’S CATCH THE CULPRIT A NEOS PERFORMANCE DETECTIVE STORY SEBASTIAN HELZLE - NEOSCON KARLSRUHE 2024
2. A NEOS PERFORMANCE DETECTIVE STORY - ABOUT ME ABOUT ME ▸ Consultant for Neos CMS projects ▸ Neos core team member ▸ 🏡 near Karlsruhe, Germany with my little family ▸ ❤ for baking, Lego, gaming ▸ @sebobo@neos.social ▸ @sebobo on Github, Slack, etc.
3. A NEOS PERFORMANCE DETECTIVE STORY - ABOUT ME ABOUT MY WORK SELF ▸ Master detective ▸ Feared by many ▸ Fluent in various programming languages ▸ Type safe ▸ Been to dark places of the core and survived ▸ „die“ is a word I often write
4. A NEOS PERFORMANCE DETECTIVE STORY - AGENDA AGENDA ▸ Case 1 - The horror of publishing ▸ Case 2 - The tale of reading image widths ▸ Case 3 - The madness of moving nodes ▸ Who is our „enemy“? ▸ What can we learn from this ▸ Notes ▸ Questions
5. Case 1 - The horror of publishing
6. CASE 1 - THE HORROR OF PUBLISHING THE CRIME SCENE ▸ Workspaces with 1000s of dependent changes ▸ Publishing could take up to 4-6 hours ▸ Was done at night 🌚 ▸ With people waking up to check the status… ⏰ ▸ Errors were super scary 😰 ▸ Little nodes could lose their parents and get orphaned ▸ GBs of memory consumption
7. CASE 1 - THE HORROR OF PUBLISHING THE SUSPECT ▸ The „old“ content repository ▸ Fires many signals ▸ Triggers lots of operations
8. CASE 1 - THE HORROR OF PUBLISHING THE ANALYSIS ▸ Make slow requests reproducible ▸ Copy as CURL from browser ▸ Run in some tool like RapidAPI (https://paw.cloud/) ▸ Found huge number of calls with Plumber and xdebug (https://github.com/sandstorm/Plumber)
9. CASE 1 - THE HORROR OF PUBLISHING THE CULPRIT ▸ Flushing the cache! ▸ Millions of single requests ▸ Thousands of unnecessary or deprecated tags ▸ Uploading & recompiling Redis scripts over and over
10. CASE 1 - THE HORROR OF PUBLISHING THE SOLUTION ▸ Check what to fl ush ▸ Ignore stale workspaces (many) ▸ Skip deprecated tags ▸ Flush in batches ▸ Analyse & optimise con fi g dynamically ▸ Only compile scripts once (Redis) ▸ Send as few requests as possible ▸ Optimise cache backend code
11. CASE 1 - THE HORROR OF PUBLISHING Before After Improvement Number of changed nodes 781 781 0,0 % Number of cache tags 463.699 4.987 98,9 % Flush cache request duration in ms 23.427 4.963 78,8 % Preparing list of cache tags in ms 2.765 65 97,6 % Flushing all tags in ms 16.362 19 99,9 %
12. CASE 1 - THE HORROR OF PUBLISHING THE OUTCOME ▸ Several minutes instead of 4-6h to publish 1000s of changes ▸ No big preparation required anymore ▸ People can sleep at night 😴 ▸ Less chance for errors (and orphans!) ▸ The PDO and Redis cache became super fast 🏎 ▸ Filesystem cache got a bit better but still a 🐌 ▸ Fixed in Neos 8.0 ▸ All PRs (https://github.com/orgs/neos/projects/30)
13. Case 2 - The tale of reading image widths
14. CASE 2 - THE TALE OF READING IMAGE WIDTHS THE CRIME SCENE ▸ Page rendering became really slow over time in production ▸ Several minutes (uncached) ▸ Smaller pages were faster ▸ Backend was generally fi ne ▸ Nobody knew what’s going on for quite a while ▸ We couldn’t analyse the production system ▸ Servers weren’t busy
15. CASE 2 - THE TALE OF READING IMAGE WIDTHS THE SUSPECTS ▸ Again the „old“ content repository 😕 ▸ The complexity of the application ⚙ ▸ „Neos“ 💔 ▸ The developers 👨💻 ▸ „The cloud“ 🌩
16. CASE 2 - THE TALE OF READING IMAGE WIDTHS THE ANALYSIS ▸ Use Flowpack.Fusion.Tracing (https://github.com/ Flowpack/Flowpack.Fusion.Tracing) ▸ In production with a trick 😉 ▸ Use Xdebug pro fi ler (http://xdebug.org/docs/pro fi ler) ▸ In development ▸ Compare local and remote traces ▸ t3n/neos-debug (https://github.com/t3n/neos-debug)
17. FUSION FLAME GRAPH
19. THIS IS IT!
20. CASE 2 - THE TALE OF READING IMAGE WIDTHS THE CULPRIT ▸ An Eel helper ▸ Read the width of an image ▸ From the fi le itself ▸ Freshly copied from S3 each time ▸ Very old piece of code ▸ Didn’t cause issues when project was smaller ▸ An each page rendering had ~13.000.000 method calls
21. CASE 2 - THE TALE OF READING IMAGE WIDTHS THE SOLUTION ▸ Read the image width from the database 🥳 ▸ Optimise some nodetype related code which caused millions of method calls ▸ Improve PHP con fi g related to the PHP JIT
22. CASE 3 - THE MADNESS OF MOVING NODES THE OUTCOME ▸ 3-7s instead of 200s to load a large page ▸ 10x improvement in loading times on average ▸ ~ 10% from the smaller optimisations ▸ Really happy editors 🥳
23. Case 3 - The madness of moving nodes
24. CASE 3 - THE MADNESS OF MOVING NODES THE CRIME SCENE ▸ We just want to move a few hundred nodes in the page tree ▸ Took minutes
25. CASE 3 - THE MADNESS OF MOVING NODES THE SUSPECT ▸ Once again the „old“ content repository 😃
26. CASE 3 - THE MADNESS OF MOVING NODES THE ANALYSIS ▸ Created pro fi les with Sandstorm.Plumber ▸ Huge number of calls in the doctrine persistence layer https://github.com/sandstorm/Plumber
27. CASE 3 - THE MADNESS OF MOVING NODES THE CULPRIT ▸ The content repository‼ & doctrine ▸ Nodes were persisted individually ▸ Triggered a n^2 cascade of calculations in doctrine
28. CASE 3 - THE MADNESS OF MOVING NODES THE SOLUTION ▸ Don’t persist each node by itself ▸ Persist all moved nodes with one call instead ▸ Doctrine only calculates actual changes once ▸ More details here (https://github.com/neos/neos- development-collection/pull/3015)
29. CASE 3 - THE MADNESS OF MOVING NODES THE OUTCOME ▸ 8s to move 750 nodes instead of 100s ▸ >10x improvement! 🐎 ▸ Fixed in Neos 4.3
30. Wo is our „Moriarty„?
31. WE ARE!
32. WO IS OUR „MORIARTY„? GENERAL ISSUES ▸ Databases handled millions of entities decades ago ▸ Probably on servers 100x slower than now ▸ We pile layers of abstractions ▸ We rarely work with low level apis ▸ Our hardware is too fast ▸ Our applications grow to complex ▸ Every environment is di ff erent ▸ Our tests are too simple
33. WO IS OUR „MORIARTY„? SOME SPECIFIC ISSUES ▸ Deep Fusion hierarchies / too much complexity ▸ Ine ffi cient custom helpers ▸ Ine ffi cient FlowQueries ▸ Requesting/calculating the same data over and over ▸ Missing or insu ffi cient Fusion cache con fi gs ▸ Using fi le based cache backends
34. What can we learn from this
35. ONCE UPON A TIME (~24 YEARS AGO)
36. WHAT CAN WE LEARN FROM THIS SOME THINGS I LEARNED SINCE THEN ▸ Premature optimisation can be a great teacher ▸ Optimising MIPS assembler code at uni taught me a lot ▸ For every great performance fi x, several were negligible 🤷 ▸ Bad performance is a 🐞 ▸ Default con fi gurations often don’t scale ▸ Improvements seem so obvious afterwards ▸ Web stu ff has no reason to be slow ▸ We basically concatenate some strings 🙄
37. WHAT CAN WE LEARN FROM THIS HOW CAN WE IMPROVE OUR PROJECTS ▸ Write native db queries ▸ The Neos 9 core goes more into that direction ▸ Learn about your database ▸ Use the right cache for the right job ▸ Reduce complexity ▸ Remember that every LOC has a cost ▸ Learn about your stack ▸ Monitor external data access ▸ Use the tools (e.g. browser, neos-debug, pro fi ler, tracer, …)
38. WO IS OUR „MORIARTY„? SOME OPTIMISATIONS ARE NEGLIGIBLE ▸ Compiler often better than we are ▸ Latency has a huge impact ▸ Only a ff ect a speci fi c environment
39. The new neos/debug plugin Planned for mid 2024 https://github.com/Sebobo/neos-debug
40. WHAT CAN WE LEARN FROM THIS WHAT DOESN’T WORK ▸ Fixing performance just by „cleaning up“ * ▸ Not reporting performance issues ▸ Hoping „someone“ will fi x it * ▸ Just using more hardware ▸ Blaming someone * Most of the time
41. WHAT CAN WE LEARN FROM THIS SUSTAINABILITY ▸ Performance is User Experience 🥰 ▸ Performance is Developer Experience 👩💻 ▸ Performance is green 🌳 ▸ Performance sells 💰 ▸ Open Source makes the di ff erence!
42. Notes
43. NOTES SOME OF MY OTHER CASES ▸ Neos PRs ▸ Slow backend with encoded non-image assets in properties (https://github.com/neos/neos-development-collection/pull/5007) ▸ Speed up node move actions (https://github.com/neos/neos-development-collection/pull/3015) ▸ Slow caches when querying nodes by related entities (https://github.com/neos/neos-development-collection/pull/4877) ▸ Load all image thumbnails at once to save lots of queries (https://github.com/neos/neos-development-collection/pull/4624) ▸ Reduce and cache nodetype schema size (https://github.com/neos/neos-development-collection/pull/4561) and (https://github.com/neos/ neos-development-collection/pull/3563) and (https://github.com/neos/neos-development-collection/pull/2887) ▸ Flush multiple caches at once (https://github.com/neos/neos-development-collection/pull/3631) ▸ Don't fl ush other users con fi guration after login (https://github.com/neos/neos-development-collection/pull/3564) ▸ Skip unnecessary queries for nodes (https://github.com/neos/neos-development-collection/pull/3357) and (https://github.com/neos/neos- development-collection/pull/3375) ▸ Any many more smaller ones ▸ Flow ▸ Postgres doesn't like distinct clauses when counting entities https://github.com/neos/ fl ow-development-collection/pull/3140 ▸ Improvements to various cache backends ▸ Neos UI
44. NOTES BIG ❤ AND THX ▸ My reviewers & testers ▸ My co-debuggers, rubber-🦆 and debug-🧸 ▸ Companies who sponsored the bug fi xes for the core ▸ Every developer who makes applications faster
45. NOTES ⚠ ADVERTISEMENT ⚠ ▸ Bad performance in Neos projects can be improved ▸ I can probably help - get in touch 🙋 ▸ Read my blogpost „The basic recipe for a fast Neos CMS website“ (https://mind-the-seb.de/blog/the-basic-recipe- for-a-fast-neos-cms-website) ▸ Check out my website (www.helzle.it) ▸ Which I planned to relaunch until the Con 🤪
46. THX!
47. QUESTIONS! QUESTIONS? ▸ Also love to hear from you after the presentation ▸ Join #performance on slack.neos.io