The Long Journey Of Making Composer Memory-efficient and Fast

Talk about the history of PHP's package manager Composer. My personal story and lots of technical insights.

I gave this talk on April 28th as part of NeosCon 2022 in Dresden, Germany.

The talk was recorded and can be seen on Youtube.

Transcript

1. The Long Journey Of Making Composer Memory-ef fi cient and Fast Neos Conference 2022
 Yanick Witschi @to fl ar
2. Hi • Yanick, 33, Swiss • I do PHP stu ff in my own company called terminal42 gmbh • I am one of the 8 Contao Open Source CMS Core Developers • I attended NeosCon 2020 and was a Neos Award Jury member in 2021! • I love astrophysics, whisky, basketball and music
3. Why me? 🤯
4. Rules are the problem! These are Rule objects.
 1 object = a few bytes
5. Composer • Finding the best solution where only one version of every package is installed, is the main task of Composer (composer update) • Researched in mathematics as „Boolean satis fi ability problem“ • Solved using a SAT solver • Composer’s SAT Solver is based on openSUSE's Libzypp* * https://en.opensuse.org/openSUSE:Libzypp_satsolver
6. SAT Solver • Most operate on the Conjunctive Normal Form • One big AND statement that consists of OR statements • (a OR b) AND (a OR b OR c) AND (a OR -b OR -c) … • (a ∨ b) ∧ (a ∨ b ∨ c) ∧ (a ∨ ¬b ∨ ¬c) • Question: What assignments for the literals (a, b, c) are needed so that the AND statement resolves to true? • There are annual competitions „Literal“ „Clause“
7. SAT Solver • A = pizza/margherita • B = toppings/mozzarella in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ … (-A || B || C) && (-A || D || E) && …
8. Terminology • Solver • Package (literal)
 pizza/margherita in 1.0.0 • Rule (clause)
 pizza/margherita in 1.0.0 requires toppings/mozzarella in 3.0.0 • Repository
 packagist.org, etc. - provides Package instances • Pool
 Collection of all Package instances
9. Composer 1 design issues • No clear separation between building the pool and the rules. Both were built at the same time. Tried to lazy load rules. • Loaded all package versions into the pool.
10. It got worse over time • Composer basically became a victim of its own success • Complexity grew with every new release of every package in your dependency tree • Complexity grew through collaboration (= more dependencies) which was now easier than ever
11. Why you no fi x in Composer 1? • There are over 1500 Composer plugins* and Composer provides a Plugin API. • Neos has one too! * https://packagist.org/packages/list.json?type=composer-plugin
12. Composer 2 • Better protocol between Composer and Packagist • Parallel fi le downloads (both metadata as well as packages) • Parallel unzipping of packages • Compiled constraint evaluations • … • Major improvements in dependency resolution
13. Resolution improvements • All improvements share the same goal:
 Reduce the total number of Rule instances • 3 key areas: • MultiCon fl ictRule (Composer 2.0) • PoolBuilder (Composer 2.0) • PoolOptimizer (Composer 2.2)
14. „require“ rules • A = pizza/margherita • B = toppings/mozzarella in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ … There is one rule per require statement 👍
15. „con fl ict“ rules • A = pizza/margherita • B = toppings/mozzarella in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ ¬B) ∧ (¬A ∨ ¬C) ∧ (¬A ∨ ¬D) ∧ (¬A ∨ ¬E) ∧ … There are as many 2 literal rules per con fl ict statement as there are packages! 😰 (¬A ∨ ¬B ∨ ¬C) ∧ (¬A ∨ ¬D ∨ ¬E) ∧ … (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ …
16. Self-referencing con fl icts • How does Composer ensure it does not install multiple versions of the same package? E.g. toppings/fresh-basil in version 2.0.0 as well as 2.0.1? • Yes! Con fl icts! Every version con fl icts with every other version! • 500 package versions result in 124 750 rules! 😰
17. MultiCon fl ictRule • Takes advantage of self-referencing con fl icts • It’s impossible for one version to con fl ict with just one of the others. They all con fl ict with each other. Always! • Represents all of them in one single rule per package! • Requires special handling in the Solver • Massive reduction in rules = massive performance improvement 🎉
18. MultiCon fl ictRule 🎉
19. PoolBuilder • Separates the Pool and Rule building process • Only loads the packages referenced (=matching constraint) anywhere in the dependency tree. • Required lots of foundation work by Jordi and Nils on e.g. composer/semver • Massive reduction in packages = less rules = massive performance improvement 🎉
20. PoolBuilder 🎉
21. 2.0, we’re way better! From 34 072 to 16 452 packages (48%) From 5 593 094 to 2 742 854 rules (49%) From 3.5GB to 1.6GB RAM (45%)
 From 88s to 39s (44%) 🎉
23. PoolOptimizer • The PHP world is lucky! Lots of packages are well maintained (= lots of bug fi x releases) • Most bug fi x releases de fi ne identical dependencies • Goal: Remove packages in the pool in order to reduce the number of rules built for them (= exponential reduction!)
24. Problem • pizza/margherita requires toppings/mozzarella in 3.* • pizza/gamberetti requires toppings/mozzarella in ^3.1 • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2 • toppings/mozzarella in our Pool: • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1
25. Problem • pizza/margherita requires toppings/mozzarella in 3.* • pizza/gamberetti requires toppings/mozzarella in ^3.1 • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2 • toppings/mozzarella in our Pool: • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1 --prefer-lowest
26. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1
27. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1
28. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1 --prefer-lowest
29. It’s more complex • Constraint compression • Aliases • Replaces • Provides • Con fl icts • …
30. Composer 2.2 Capable of fi ltering >80% 
 of all packages 🎉
31. Composer today From 34 072 to 2 864 packages (8%) From 5 593 094 to 82 422 rules (1.5%) From 3.5GB to 175MB RAM (5%)
 From 88s to 10s (11%) 3 million (!) „composer update“ per day! That’s a huge win! 🌎 🎉
32. Thank you • Twitter: @to fl ar • GitHub: to fl ar • E-Mail: yanick@terminal42.ch • Slack • Come get a sticker! 🇺🇦 ❤ 🕊