The Long Journey Of Making Composer Memory-efficient and Fast

Talk about the history of PHP's package manager Composer. My personal story and lots of technical insights.

I gave this talk on April 28th as part of NeosCon 2022 in Dresden, Germany.

The talk was recorded and can be seen on Youtube.

Transcript

1. The Long Journey Of
   
   
   Making Composer
   
   
   Memory-ef
   fi
   cient
   
   
   and Fast
   Neos Conference 2022

   Yanick Witschi
   
   @to
   fl
   ar
   

   View Slide

2. Hi
   • Yanick, 33, Swiss
   
   • I do PHP stu
   ff
   in my own company called terminal42 gmbh
   
   • I am one of the 8 Contao Open Source CMS Core
   Developers
   
   • I attended NeosCon 2020 and was a Neos Award Jury
   member in 2021!
   
   • I love astrophysics, whisky, basketball and music
   

   View Slide

3. Why me?
   🤯
   

   View Slide

4. Rules are the problem!
   These are Rule objects.

   1 object = a few bytes
   

   View Slide

5. Composer
   • Finding the best solution where only one version of every package
   is installed, is the main task of Composer (composer update)
   
   • Researched in mathematics as „Boolean satis
   fi
   ability problem“
   
   • Solved using a SAT solver
   
   • Composer’s SAT Solver is based on openSUSE's Libzypp*
   * https://en.opensuse.org/openSUSE:Libzypp_satsolver
   

   View Slide

6. SAT Solver
   • Most operate on the Conjunctive Normal Form
   
   • One big AND statement that consists of OR statements
   
   • (a OR b) AND (a OR b OR c) AND (a OR -b OR -c) …
   
   • (a ∨ b) ∧ (a ∨ b ∨ c) ∧ (a ∨ ¬b ∨ ¬c)
   
   • Question: What assignments for the literals (a, b, c) are
   needed so that the AND statement resolves to true?
   
   • There are annual competitions
   „Literal“
   „Clause“
   

   View Slide

7. SAT Solver
   • A = pizza/margherita
   
   • B = toppings/mozzarella in version 3.0.0
   
   • C = toppings/mozzarella in version 3.1.0
   
   • D = toppings/fresh-basil in version 2.0.0
   
   • E = toppings/fresh-basil in version 2.0.1
   (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ …
   (-A || B || C) && (-A || D || E) && …
   

   View Slide

8. Terminology
   • Solver
   
   • Package (literal)

   pizza/margherita in 1.0.0
   
   • Rule (clause)

   pizza/margherita in 1.0.0 requires toppings/mozzarella in 3.0.0
   
   • Repository

   packagist.org, etc. - provides Package instances
   
   • Pool

   Collection of all Package instances
   

   View Slide

9. Composer 1 design issues
   • No clear separation between building the pool and the
   rules. Both were built at the same time. Tried to lazy load
   rules.
   
   • Loaded all package versions into the pool.
   

   View Slide

10. It got worse over time
    • Composer basically became a victim of its own success
    
    • Complexity grew with every new release of every package in your
    dependency tree
    
    • Complexity grew through collaboration (= more dependencies)
    which was now easier than ever
    

    View Slide

11. Why you no
    fi
    x in
    Composer 1?
    • There are over 1500 Composer plugins* and Composer
    provides a Plugin API.
    
    • Neos has one too!
    * https://packagist.org/packages/list.json?type=composer-plugin
    

    View Slide

12. Composer 2
    • Better protocol between Composer and Packagist
    
    • Parallel
    fi
    le downloads (both metadata as well as packages)
    
    • Parallel unzipping of packages
    
    • Compiled constraint evaluations
    
    • …
    
    • Major improvements in dependency resolution
    

    View Slide

13. Resolution improvements
    • All improvements share the same goal:

    Reduce the total number of Rule instances
    
    • 3 key areas:
    
    • MultiCon
    fl
    ictRule (Composer 2.0)
    
    • PoolBuilder (Composer 2.0)
    
    • PoolOptimizer (Composer 2.2)
    

    View Slide

14. „require“ rules
    • A = pizza/margherita
    
    • B = toppings/mozzarella in version 3.0.0
    
    • C = toppings/mozzarella in version 3.1.0
    
    • D = toppings/fresh-basil in version 2.0.0
    
    • E = toppings/fresh-basil in version 2.0.1
    (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ …
    There is one rule per require
    statement 👍
    

    View Slide

15. „con
    fl
    ict“ rules
    • A = pizza/margherita
    
    • B = toppings/mozzarella in version 3.0.0
    
    • C = toppings/mozzarella in version 3.1.0
    
    • D = toppings/fresh-basil in version 2.0.0
    
    • E = toppings/fresh-basil in version 2.0.1
    (¬A ∨ ¬B) ∧ (¬A ∨ ¬C) ∧ (¬A ∨ ¬D) ∧ (¬A ∨ ¬E) ∧ …
    There are as many 2 literal rules
    per con
    fl
    ict statement as there
    are packages! 😰
    (¬A ∨ ¬B ∨ ¬C) ∧ (¬A ∨ ¬D ∨ ¬E) ∧ …
    (¬A ∨ B ∨ C) ∧ (¬A ∨ D ∨ E) ∧ …
    

    View Slide

16. Self-referencing con
    fl
    icts
    • How does Composer ensure it does not install multiple
    versions of the same package? E.g. toppings/fresh-basil in
    version 2.0.0 as well as 2.0.1?
    
    • Yes! Con
    fl
    icts! Every version con
    fl
    icts with every other
    version!
    
    •
    500 package versions result in
    124 750 rules! 😰
    

    View Slide

17. MultiCon
    fl
    ictRule
    • Takes advantage of self-referencing con
    fl
    icts
    
    • It’s impossible for one version to con
    fl
    ict with just one of the
    others. They all con
    fl
    ict with each other. Always!
    
    • Represents all of them in one single rule per package!
    
    • Requires special handling in the Solver
    
    • Massive reduction in rules = massive performance
    improvement 🎉
    

    View Slide

18. MultiCon
    fl
    ictRule 🎉
    

    View Slide

19. PoolBuilder
    • Separates the Pool and Rule building process
    
    • Only loads the packages referenced (=matching constraint)
    anywhere in the dependency tree.
    
    • Required lots of foundation work by Jordi and Nils on e.g.
    composer/semver
    
    • Massive reduction in packages = less rules = massive
    performance improvement 🎉
    

    View Slide

20. PoolBuilder 🎉
    

    View Slide

21. 2.0, we’re way better!
    From 34 072 to 16 452 packages (48%)
    
    From 5 593 094 to 2 742 854 rules (49%)
    
    From 3.5GB to 1.6GB RAM (45%)

    From 88s to 39s (44%)
    🎉
    

    View Slide

22. View Slide

23. PoolOptimizer
    • The PHP world is lucky! Lots of packages are well
    maintained (= lots of bug
    fi
    x releases)
    
    • Most bug
    fi
    x releases de
    fi
    ne identical dependencies
    
    • Goal: Remove packages in the pool in order to reduce the
    number of rules built for them (= exponential reduction!)
    

    View Slide

24. Problem
    • pizza/margherita requires toppings/mozzarella in 3.*
    
    • pizza/gamberetti requires toppings/mozzarella in ^3.1
    
    • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2
    
    • toppings/mozzarella in our Pool:
    
    • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1
    

    View Slide

25. Problem
    • pizza/margherita requires toppings/mozzarella in 3.*
    
    • pizza/gamberetti requires toppings/mozzarella in ^3.1
    
    • pizza/quattro-stagioni requires toppings/mozzarella in ^3.2
    
    • toppings/mozzarella in our Pool:
    
    • 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0, 3.3.1
    --prefer-lowest
    

    View Slide

26. Dependency groups
    • toppings/mozzarella in ^3.1
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in 3.*
    
    • 3.0.0
    
    • 3.0.1
    
    • 3.0.2
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in ^3.2
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    

    View Slide

27. Dependency groups
    • toppings/mozzarella in ^3.1
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in 3.*
    
    • 3.0.0
    
    • 3.0.1
    
    • 3.0.2
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in ^3.2
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    

    View Slide

28. Dependency groups
    • toppings/mozzarella in ^3.1
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in 3.*
    
    • 3.0.0
    
    • 3.0.1
    
    • 3.0.2
    
    • 3.1.0
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    • toppings/mozzarella in ^3.2
    
    • 3.2.0
    
    • 3.3.0
    
    • 3.3.1
    --prefer-lowest
    

    View Slide

29. It’s more complex
    • Constraint compression
    
    • Aliases
    
    • Replaces
    
    • Provides
    
    • Con
    fl
    icts
    
    • …
    

    View Slide

30. Composer 2.2
    Capable of
    fi
    ltering >80%
    

    of all packages
    🎉
    

    View Slide

31. Composer today
    From 34 072 to 2 864 packages (8%)
    
    From 5 593 094 to 82 422 rules (1.5%)
    
    From 3.5GB to 175MB RAM (5%)

    From 88s to 10s (11%)
    3 million (!) „composer update“
    
    per day! That’s a huge win!
    
    🌎 🎉
    

    View Slide

32. Thank you
    • Twitter: @to
    fl
    ar
    
    • GitHub: to
    fl
    ar
    
    • E-Mail: yanick@terminal42.ch
    
    • Slack
    
    • Come get a sticker!
    🇺🇦 ❤ 🕊
    

    View Slide