Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Feedback on scalability and load testing of a configuration management software

7d9785e3bdceb2d9e86dabcfb77b1686?s=47 Rudder
February 04, 2020

Feedback on scalability and load testing of a configuration management software

🎥 https://www.youtube.com/watch?v=rQlLzVbXEXk
🧑 Nicolas Charles
📅 Configuration Management Camp 2020

RUDDER is based on API/Web application that allows users to configure and verify their configurations. Relying on agents on every system, it checks and remediates configurations every 5 minutes and centralizes the result of application. Each result is made up of hundreds of events that are historized, and each configuration changes involves calculating and displaying the configurations and conformities for users within a reasonable time.

A RUDDER instance can handle 20 000 nodes. Can you imagine what this implies from a network, CPU and storage point of view? How to reach and maintain these performances? What are the different steps that made this possible? And what tools have been put in place?

This presentation will explain the technical stack used (Scala, PostgreSQL, C and Rust), as well as the path, failures and successes that allow us today to reproduce the environments, and also to test and validate the hypotheses to achieve and keep these results.

7d9785e3bdceb2d9e86dabcfb77b1686?s=128

Rudder

February 04, 2020
Tweet

Transcript

  1. rudder.io Feedback on scalability and load testing of a configuration

    management software Nicolas CHARLES nicolas@rudder.io - @nico_charles
  2. • RUDDER is a Continuous Configuration Management software with a

    Compliance focus • Configurations are defined with GUI or API • Configuration checked on every nodes every 5 minutes • RUDDER computes the compliance of every nodes RUDDER - Why scalability is a topic?
  3. RUDDER - Defining configurations

  4. RUDDER - Defining configurations PARAM RULE • Id DIRECTIVE •

    Id • (Components) GROUP • Id RUDDER config (global) • Policy Mode • Schedule NODE • Properties • Policy Mode • Schedule Environmental context • Id : . . . • Generated : . . . Files Node configuration Historization Historization
  5. RUDDER - Computing compliance 5 • Id : . .

    . • Generated : . . . Files Node configuration RUN • Reports • Reports • ... • ... METADATA • node id • config id • run timestamp RUN • Reports • Reports • ... • ... METADATA • node id • config id • run timestamp • Signature Get Policy Send configuration reports Expected reports (node id, config id, timestamp) Run reports Historization Compliance (historized) Store expected reports Metadata • Integrity • Signature Config • Id • For Rule R, Directive D1, Component C On every nodes
  6. RUDDER - Compliance

  7. • Load testing is the process of putting demand on

    a system and measuring its response.[2] • Simulating real-world usage, and ensuring the system can sustain it. Load testing? [2] https://en.wikipedia.org/wiki/Load_testing
  8. • For SaaS systems, you manage the lifecycle of your

    application, so you can test for top of next months expected scale. • For a software deployed on users infrastructures, upgrades are done during maintenance windows ◦ They need (long) migration tests ⇨ You have to think forward Testing at what scale??
  9. • A German cars manufacturer manages 13 000 nodes with

    a 3 years-old RUDDER 3.1 instance, and 4 500 nodes with a 2 years old RUDDER 4.1 version • Need to consolidate on one instance • Expect 20 000 nodes this year How hard can it be? RUDDER - Testing at what scale?
  10. • RUDDER supports “old” OS - RHEL 7, SLES 12

    • Uses system dependencies, which can be antediluvian: PostgreSQL 9.2, rsyslog 8.24.0 ◦ Their performance can be problematic RUDDER - Some constraints
  11. RUDDER - architecture Rudder Root Server Interfaces CLI WEB UI

    API Users Applications Compliance Configuration Inventory Rudder Engine + Plugins Relay Node Rudder Agent Node Rudder Agent Node Rudder Agent
  12. RUDDER - components Rudder Root Server Interfaces CLI WEB UI

    API Users Applications Compliance Configuration Inventory Rudder Engine + Plugins Relay Node Rudder Agent Node Rudder Agent Node Rudder Agent Scala PostgreSQL OpenLDAP rsyslog
  13. RUDDER - where could be the bottlenecks? Rudder Root Server

    Interfaces CLI WEB UI API Users Applications Compliance Configuration Inventory Rudder Engine + Plugins Relay Node Rudder Agent Node Rudder Agent Node Rudder Agent
  14. • Testing every parts of Rudder ◦ In isolation first,

    then all together ◦ Ramp-up the load • Inventories • Set of groups, techniques, directives , rules • Reports from all nodes • Relevant logs and way to analyze them Testing for 20 000 nodes - what is needed?
  15. • Script to generate inventories ◦ Template + Tsung, contributed

    by community https://github.com/Normation/rudder-tools/tree/master/contrib/stress_suite ◦ Bash scripts with more flexibility: generated certificates, multiple templates, etc; plus persistent data to generate reports https://github.com/Normation/rudder-tools/pull/534/files Testing for 20 000 nodes - Tooling?
  16. • Script to generate reports from created nodes ◦ Python

    script to get expected reports from the database, generate and send reports over syslog https://github.com/Normation/rudder-tools/tree/master/contrib/load-database ◦ Python script to get expected reports from the database, generates reports, sign them using the static data and sends them over HTTPS https://github.com/Normation/rudder-tools/pull/534/files Testing for 20 000 nodes - Tooling?
  17. • First test, with 6000 nodes • Should be easy,

    as earlier versions of RUDDER supported 13000 nodes… Testing for 20 000 nodes
  18. • First test, with 6000 nodes • Should be easy,

    as earlier versions of RUDDER supported 13000 nodes… • Added features in 5.0 have a significant cost in scalability :( Testing for 20 000 nodes
  19. • The most resource intensive part a. Snapshot inventories, groups,

    directive and rules b. Historize data c. Convert them in policies mapping d. Generate files with data and policies for the nodes e. Save expected reports f. Run post-generation hooks RUDDER - Policy generation
  20. • Extract timings ◦ Each step of Policy Generation is

    measured… but Done XXX in: 322 350 ms is clearly not precise enough ◦ Need more detailed timings on each parts ▪ Aggregate timings over each threads and components • Process timings ◦ Automate extracting features from thousands of measures ▪ 100 000 message “mergeCompareByRules in x ms” RUDDER - Policy generation
  21. • Inspect slow parts ◦ Variable computation ◦ Database reads/writes

    ◦ I/O • Ensure that memory usage is reasonable • The core of RUDDER is written in Scala, so there is a lot of available tooling RUDDER - Policy generation
  22. • jmap (fast) ◦ histogram of all objects allocated, memory

    used and number ◦ dump of JVM state RUDDER - JVM tooling
  23. • Eclipse Memory Analyzer (async) ◦ analyse dump and suggest

    fixes ◦ doesn’t work great for large heap RUDDER - JVM tooling
  24. • YourKit ◦ Instrument the JVM ◦ Measure object creation,

    methods calls, memory used, etc ◦ Great detailed vision RUDDER - JVM tooling
  25. • YourKit ◦ Instrument the JVM ◦ Measure object creation,

    methods calls, memory used, etc ◦ Great detailed vision of systems ◦ But massive slowdown of the application when activated RUDDER - JVM tooling
  26. • YourKit ◦ Instrument the JVM ◦ Measure object creation,

    methods calls, memory used, etc ◦ Great detailed vision of systems ◦ But massive slowdown of the application when activated RUDDER - JVM tooling
  27. • https://gceasy.io/ ◦ Analyze GC logs and extract metrics and

    health of GC RUDDER - JVM tooling
  28. • Facts ◦ Too many objects are created with super

    short live ▪ Strain on GC, increase CPUs usage ◦ Some unnecessary computations ◦ Serializing and deserializing JSON is CPU and Memory expensive ◦ IO is super expensive RUDDER - Policy generation
  29. • Optimizations ◦ Use collection views when necessary ▪ Caution,

    sometimes views are hidden and cause leaks ◦ Fix some algorithms to remove unnecessary lookups ◦ Identify the necessary lifespan of big objects, to ensure there are not referenced when not necessary ◦ Batched all requests to database ▪ Lower memory usage ▪ Big strings from PostgreSQL that are deserialized in one go RUDDER - Policy generation
  30. • Python script to generate reports • 3000+ syslog messages/second

    • Catastrophic compliance RUDDER - Reports and compliance
  31. • Python script to generate reports • 3000+ syslog messages/second

    • Catastrophic compliance • nestat -suna ◦ 80% of messages are dropped... RUDDER - Reports and compliance
  32. • rsyslog inserts report in PostgreSQL 1 by 1 ◦

    Need to upgrade to 2019 versions • Increase UDP buffer to handle the load • Increase the number of workers https://docs.rudder.io/reference/5.0/administration/performance.html#_increase_th e_udp_buffer • Fixed the network part for now RUDDER - Reports and compliance
  33. • rsyslog inserts report in PostgreSQL 1 by 1 ◦

    Need to upgrade to 2019 versions • Increase UDP buffer to handle the load • Increase the number of workers https://docs.rudder.io/reference/5.0/administration/performance.html#_increase_th e_udp_buffer • Fixed the network part for now ◦ In 6.0, we dropped rsyslog, to use HTTPS reports parsed by a Rust module RUDDER - Reports and compliance
  34. • Compliance computation can’t deal with all reports • Process

    for computation ◦ Fetch nodes last runs ◦ Get expected reports ◦ Get reports from these nodes ◦ Expand variables, and compute compliance • Depends on PostgreSQL • Computations and lookups on the Scala part RUDDER - Reports and compliance
  35. • PgBadger ◦ Analyze PostgreSQL logs ◦ Very complete stats

    on database usage ▪ Top queries, most expensive queries, temp files RUDDER - Tooling for PostgreSQL
  36. • https://explain.depesz.com/ ◦ Explain query plan ◦ In an understandable

    way ◦ Help correct the queries RUDDER - Tooling for PostgreSQL
  37. • Give enough resources to PostgreSQL to handle the load

    ◦ shared_buffer ◦ work_mem ◦ maintenance_work_mem ◦ temp_buffers ◦ wal_buffer • Configure it based on the reality of the system ◦ random_page_cost is too large for SSD by default ◦ effective_cache_size RUDDER - Configure PostgreSQL
  38. • Optimizations in the Scala part ◦ Improvement in variables

    detections ◦ Lighter data structures RUDDER - Reports and compliance
  39. • Options to disable features to save resources ◦ Full

    historization of compliance in database ◦ Historization of non-compliance reports ◦ Changes computation ◦ Recompute of dynamics groups RUDDER - Feature flags
  40. • Define timeouts, parallelism... RUDDER - Tunability

  41. • Load testing is time consuming ◦ 10 minutes to

    12 hours for a relevant test ◦ Enormous logs ▪ Scripts to extracts stats on these logs automatically ◦ Analyzing the results can be tedious ▪ Especially as I/O can be erratics (sometimes *4 perf degradation) • Generating relevant data is hard ◦ A German cars manufacturer gave us access to a system with their inventories to have real data - thank you ! RUDDER - Load Testing Takeaway
  42. • Beware of intuitions ◦ Guesses and hypothesis are often

    wrong ▪ Need to be meticulous ◦ Some code or may look bad, but turn out to be pretty efficient • You need to be persistent ◦ Only after looking many times at the same code did I see potential issues. ◦ Some changes needed a lot of thinking and interaction with other developers ▪ Thank you for bearing with me!! RUDDER - Load Testing Takeaway
  43. • Sometimes, making things faster makes them slower ◦ As

    they starve other resources ◦ Policy generation suffered from improved compliance computation • Libraries/dependencies have a real impact on performance ◦ Our JSON serializing/deserializing library is 8 years old, and really inefficient ◦ Templating engine library is not thread-safe ◦ Upgrading rsyslog, postgresql and java helped a lot RUDDER - Load Testing Takeaway
  44. • JVM JIT optimizes a lot of stuff ◦ Some

    optimizations failed to change anything ◦ JIT optimized the hell out of it ◦ Unit tests show massive perf improvement over each execution iteration RUDDER - Load Testing Takeaway
  45. • Some changes are simply too large for a minor

    version ◦ Next optimization can only fit in a major version: ▪ stream from/to database ▪ rewrite algorithms ▪ change serialization/deserialization librairies ▪ change templating engine ▪ generate policies node by node RUDDER - Load Testing Takeaway
  46. • Improving scalability in one point shows the next bottlenecks

    ◦ It can feel like a never ending task ◦ But they get smaller and smaller RUDDER - Load Testing Takeaway
  47. • Results ◦ 28 PRs for performance in 3 latests

    minors versions of Rudder ◦ 30% improvement in memory usage ◦ 40% faster compliance computation RUDDER - Did it matter?
  48. • Policy generation time is linear in number of nodes

    and components • Memory usage is linear-ish in number of nodes and components • With complex configuration (JS variables, many components per nodes): ◦ 4500 nodes, 1,6 Million components: 10 GB, 13 minutes for policy generation ◦ 10 000 nodes, 11 Millions components: 67 GB, 40 minutes ◦ 12 000 nodes, 13,2 Millions components: 82 GB, 48 minutes • With actual german manufacturer configuration ◦ Full policy generation for 12 500 nodes in 7m40 RUDDER - Did it matter?
  49. • RUDDER is limited by available IO and Memory ◦

    It does make really good use of CPUs • With enough RAM, RUDDER supports 20k nodes RUDDER - Did it matter?
  50. • Test on platform with 1TB RAMDisk ◦ Lower cost

    of I/O • Automate all these tests and measures ◦ System is now stable enough to be automated RUDDER - What’s next?
  51. • Repeat on branch 6.0 ◦ Totally different underlying framework

    ◦ Code changed a lot between the two branches ◦ Many new features in 6.0 ◦ A lot of work has already been done, but we hope to have better perfs than in 5.0 RUDDER - What’s next?
  52. • Build with observability ◦ Gather data, measure evolution ◦

    Don’t need to add logs when an issue arise ◦ Helpful to identify performance regression ◦ • Keep the librairies up to date ◦ Evaluate new librairies regularly ◦ Enormous cost of changing deprecated librairies ◦ Huge impact in performance and quality What could have been better?
  53. rudder.io Feedback on scalability and load testing of a configuration

    management software Questions ?
  54. • Why not testing with real 20000 nodes? ◦ Mimicking

    the nodes give pertinent insights ◦ It’s cost effective (the test platform cost ~100€/month) ◦ Setting up an infra (storage, network, etc) for 20000 nodes is *really* complex and time consuming ▪ Even in the cloud ▪ And it needs to run for quite a long time The elephant in the room
  55. • Why not Gatling??? ◦ A lot of iterations and

    tests of tests ◦ Tests on several versions with different methods ◦ Need to consolidate The elephant in the room