Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Long Journey Of Making Composer Memory-effi...

The Long Journey Of Making Composer Memory-efficient and Fast

Talk about the history of PHP's package manager Composer. My personal story and lots of technical insights.

I gave this talk on April 28th as part of NeosCon 2022 in Dresden, Germany.

The talk was recorded and can be seen on Youtube.

Yanick Witschi

May 20, 2022
Tweet

More Decks by Yanick Witschi

Other Decks in Programming

Transcript

  1. The Long Journey Of Making Composer Memory-ef fi cient and

    Fast Neos Conference 2022
 Yanick Witschi @to fl ar
  2. Hi • Yanick, 33, Swiss • I do PHP stu

    ff in my own company called terminal42 gmbh • I am one of the 8 Contao Open Source CMS Core Developers • I attended NeosCon 2020 and was a Neos Award Jury member in 2021! • I love astrophysics, whisky, basketball and music
  3. Composer • Finding the best solution where only one version

    of every package is installed, is the main task of Composer (composer update) • Researched in mathematics as „Boolean satis fi ability problem“ • Solved using a SAT solver • Composer’s SAT Solver is based on openSUSE's Libzypp* * https://en.opensuse.org/openSUSE:Libzypp_satsolver
  4. SAT Solver • Most operate on the Conjunctive Normal Form

    • One big AND statement that consists of OR statements • (a OR b) AND (a OR b OR c) AND (a OR -b OR -c) … • (a ∨ b) ∧ (a ∨ b ∨ c) ∧ (a ∨ ¬b ∨ ¬c) • Question: What assignments for the literals (a, b, c) are needed so that the AND statement resolves to true? • There are annual competitions „Literal“ „Clause“
  5. SAT Solver • A = pizza/margherita • B = toppings/mozzarella

    in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ … (-A || B || C) && (-A || D || E) && …
  6. Terminology • Solver • Package (literal)
 pizza/margherita in 1.0.0 •

    Rule (clause)
 pizza/margherita in 1.0.0 requires toppings/mozzarella in 3.0.0 • Repository
 packagist.org, etc. - provides Package instances • Pool
 Collection of all Package instances
  7. Composer 1 design issues • No clear separation between building

    the pool and the rules. Both were built at the same time. Tried to lazy load rules. • Loaded all package versions into the pool.
  8. It got worse over time • Composer basically became a

    victim of its own success • Complexity grew with every new release of every package in your dependency tree • Complexity grew through collaboration (= more dependencies) which was now easier than ever
  9. Why you no fi x in Composer 1? • There

    are over 1500 Composer plugins* and Composer provides a Plugin API. • Neos has one too! * https://packagist.org/packages/list.json?type=composer-plugin
  10. Composer 2 • Better protocol between Composer and Packagist •

    Parallel fi le downloads (both metadata as well as packages) • Parallel unzipping of packages • Compiled constraint evaluations • … • Major improvements in dependency resolution
  11. Resolution improvements • All improvements share the same goal:
 Reduce

    the total number of Rule instances • 3 key areas: • MultiCon fl ictRule (Composer 2.0) • PoolBuilder (Composer 2.0) • PoolOptimizer (Composer 2.2)
  12. „require“ rules • A = pizza/margherita • B = toppings/mozzarella

    in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ … There is one rule per require statement 👍
  13. „con fl ict“ rules • A = pizza/margherita • B

    = toppings/mozzarella in version 3.0.0 • C = toppings/mozzarella in version 3.1.0 • D = toppings/fresh-basil in version 2.0.0 • E = toppings/fresh-basil in version 2.0.1 (¬A ∨ ¬B) ∧ (¬A ∨ ¬C) ∧ (¬A ∨ ¬D) ∧ (¬A ∨ ¬E) ∧ … There are as many 2 literal rules per con fl ict statement as there are packages! 😰 (¬A ∨ ¬B ∨ ¬C) ∧ (¬A ∨ ¬D ∨ ¬E) ∧ … (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ …
  14. Self-referencing con fl icts • How does Composer ensure it

    does not install multiple versions of the same package? E.g. toppings/fresh-basil in version 2.0.0 as well as 2.0.1? • Yes! Con fl icts! Every version con fl icts with every other version! • 500 package versions result in 124 750 rules! 😰
  15. MultiCon fl ictRule • Takes advantage of self-referencing con fl

    icts • It’s impossible for one version to con fl ict with just one of the others. They all con fl ict with each other. Always! • Represents all of them in one single rule per package! • Requires special handling in the Solver • Massive reduction in rules = massive performance improvement 🎉
  16. PoolBuilder • Separates the Pool and Rule building process •

    Only loads the packages referenced (=matching constraint) anywhere in the dependency tree. • Required lots of foundation work by Jordi and Nils on e.g. composer/semver • Massive reduction in packages = less rules = massive performance improvement 🎉
  17. 2.0, we’re way better! From 34 072 to 16 452

    packages (48%) From 5 593 094 to 2 742 854 rules (49%) From 3.5GB to 1.6GB RAM (45%)
 From 88s to 39s (44%) 🎉
  18. PoolOptimizer • The PHP world is lucky! Lots of packages

    are well maintained (= lots of bug fi x releases) • Most bug fi x releases de fi ne identical dependencies • Goal: Remove packages in the pool in order to reduce the number of rules built for them (= exponential reduction!)
  19. Problem • pizza/margherita requires toppings/mozzarella in 3.* • pizza/gamberetti requires

    toppings/mozzarella in ^3.1 • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2 • toppings/mozzarella in our Pool: • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1
  20. Problem • pizza/margherita requires toppings/mozzarella in 3.* • pizza/gamberetti requires

    toppings/mozzarella in ^3.1 • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2 • toppings/mozzarella in our Pool: • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1 --prefer-lowest
  21. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0

    • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1
  22. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0

    • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1
  23. Dependency groups • toppings/mozzarella in ^3.1 • 3.1.0 • 3.2.0

    • 3.3.0 • 3.3.1 • toppings/mozzarella in 3.* • 3.0.0 • 3.0.1 • 3.0.2 • 3.1.0 • 3.2.0 • 3.3.0 • 3.3.1 • toppings/mozzarella in ^3.2 • 3.2.0 • 3.3.0 • 3.3.1 --prefer-lowest
  24. Composer today From 34 072 to 2 864 packages (8%)

    From 5 593 094 to 82 422 rules (1.5%) From 3.5GB to 175MB RAM (5%)
 From 88s to 10s (11%) 3 million (!) „composer update“ per day! That’s a huge win! 🌎 🎉
  25. Thank you • Twitter: @to fl ar • GitHub: to

    fl ar • E-Mail: [email protected] • Slack • Come get a sticker! 🇺🇦 ❤ 🕊