Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to make easy and casual CI management in COOKPAD

How to make easy and casual CI management in COOKPAD

Kenta Murata

July 24, 2013
Tweet

More Decks by Kenta Murata

Other Decks in Technology

Transcript

  1. How to make easy and
    casual CI management
    in COOKPAD
    Kenta Murata
    2013.07.24
    Testing Casual Talks #1 at :DeNA
    Friday, July 26, 13

    View full-size slide

  2. Kenta Murata
    CRuby committer
    DevInfra Engineer at COOKPAD
    Friday, July 26, 13

    View full-size slide

  3. Friday, July 26, 13

    View full-size slide

  4. Friday, July 26, 13

    View full-size slide

  5. Friday, July 26, 13

    View full-size slide

  6. bit.ly/cookpad_jobs
    Friday, July 26, 13

    View full-size slide

  7. Background
    Friday, July 26, 13

    View full-size slide

  8. About 1 year ago...
    Friday, July 26, 13

    View full-size slide

  9. $ ree_be rake stats
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Controllers | 16708 | 13381 | 154 | 1153 | 7 | 9 |
    | Helpers | 6228 | 5179 | 0 | 508 | 0 | 8 |
    | Models | 38665 | 30777 | 861 | 3248 | 3 | 7 |
    | Libraries | 29363 | 23740 | 238 | 2189 | 9 | 8 |
    | Functional tests | 9 | 7 | 1 | 0 | 0 | 0 |
    | Unit tests | 72 | 54 | 10 | 0 | 0 | 0 |
    | Integration specs | 21207 | 17381 | 2 | 83 | 41 | 207 |
    | Chanko specs | 15700 | 13030 | 1 | 103 | 103 | 124 |
    | Model specs | 22247 | 18297 | 3 | 35 | 11 | 520 |
    | Controller specs | 17061 | 14080 | 6 | 49 | 8 | 285 |
    | Helper specs | 588 | 507 | 0 | 0 | 0 | 0 |
    | Routing specs | 128 | 102 | 1 | 1 | 1 | 100 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Total | 167976 | 136535 | 1277 | 7369 | 5 | 16 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    Code LOC: 73077 Test LOC: 63458 Code to Test Ratio: 1:0.9
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  10. $ ree_be rake stats
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Controllers | 16708 | 13381 | 154 | 1153 | 7 | 9 |
    | Helpers | 6228 | 5179 | 0 | 508 | 0 | 8 |
    | Models | 38665 | 30777 | 861 | 3248 | 3 | 7 |
    | Libraries | 29363 | 23740 | 238 | 2189 | 9 | 8 |
    | Functional tests | 9 | 7 | 1 | 0 | 0 | 0 |
    | Unit tests | 72 | 54 | 10 | 0 | 0 | 0 |
    | Integration specs | 21207 | 17381 | 2 | 83 | 41 | 207 |
    | Chanko specs | 15700 | 13030 | 1 | 103 | 103 | 124 |
    | Model specs | 22247 | 18297 | 3 | 35 | 11 | 520 |
    | Controller specs | 17061 | 14080 | 6 | 49 | 8 | 285 |
    | Helper specs | 588 | 507 | 0 | 0 | 0 | 0 |
    | Routing specs | 128 | 102 | 1 | 1 | 1 | 100 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Total | 167976 | 136535 | 1277 | 7369 | 5 | 16 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    Code LOC: 73077 Test LOC: 63458 Code to Test Ratio: 1:0.9
    861 Models
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  11. $ ree_be rake stats
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Controllers | 16708 | 13381 | 154 | 1153 | 7 | 9 |
    | Helpers | 6228 | 5179 | 0 | 508 | 0 | 8 |
    | Models | 38665 | 30777 | 861 | 3248 | 3 | 7 |
    | Libraries | 29363 | 23740 | 238 | 2189 | 9 | 8 |
    | Functional tests | 9 | 7 | 1 | 0 | 0 | 0 |
    | Unit tests | 72 | 54 | 10 | 0 | 0 | 0 |
    | Integration specs | 21207 | 17381 | 2 | 83 | 41 | 207 |
    | Chanko specs | 15700 | 13030 | 1 | 103 | 103 | 124 |
    | Model specs | 22247 | 18297 | 3 | 35 | 11 | 520 |
    | Controller specs | 17061 | 14080 | 6 | 49 | 8 | 285 |
    | Helper specs | 588 | 507 | 0 | 0 | 0 | 0 |
    | Routing specs | 128 | 102 | 1 | 1 | 1 | 100 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Total | 167976 | 136535 | 1277 | 7369 | 5 | 16 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    Code LOC: 73077 Test LOC: 63458 Code to Test Ratio: 1:0.9
    17381 Integration spec LOC
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  12. 6000
    6500
    7000
    7500

































    ● ●





































































































    ● ●●




























































































































    ● ●●●

























































































































    ● ●


    ●●





















































































































































    ●●
    ● ●




















































































































    ● ●






















    ●●
































































































    ● ●




































































































    ●●
    ● ●




















































































































    ● ●











































































































    ● ●






































































































    05/01 06/01 07/01
    Timestamp
    Examples count
    6000
    6500
    7000
    7500

































    ● ●





































































































    ● ●●




























































































































    ● ●●●

























































































































    ● ●


    ●●





















































































































































    ●●
    ● ●




















































































































    ● ●






















    ●●
































































































    ● ●




































































































    ●●
    ● ●




















































































































    ● ●











































































































    ● ●






































































































    05/01 06/01 07/01
    Timestamp
    Examples count
    The number of examples in
    COOKPAD’s spec les
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide






















































































































  13. ● ●











































































































    ● ●





































































































    ●●

    ● ●


























































































    ● ●








































































































    ● ●






































































































    ● ●



































    ●●








































    ● ●


















































































    ●●









    ● ●






































































    ● ●






















































    ●●











    ● ● ●













































































    ● ●






























    07/01 08/01 09/01
    Timestamp
    6000
    6500
    7000
    7500

































    ● ●





































































































    ● ●●




























































































































    ● ●●●

























































































































    ● ●


    ●●





















































































































































    ●●
    ● ●




















































































































    ● ●






















    ●●
































































































    ● ●




































































































    ●●
    ● ●




















































































































    ● ●











































































































    ● ●






































































































    05/01 06/01 07/01
    Timestamp
    Examples count
    The number of examples in
    COOKPAD’s spec les
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  14. Condition Execution time
    single rspec too long; unmeasurable
    8-core parallel_tests over 1 hour
    5 workers,
    each have 6-cores
    about 15 min
    Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  15. Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  16. Sunday, September 16, 12
    in Sapporo RubyKaigi 2012
    Friday, July 26, 13

    View full-size slide

  17. After that...
    Friday, July 26, 13

    View full-size slide

  18. $ be rake cookpad:stats
    Friday, July 26, 13

    View full-size slide

  19. $ be rake cookpad:stats
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Name | Lines | LOC | Classes | Methods | M/C | LOC/M |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Controllers | 27186 | 21690 | 257 | 2074 | 8 | 8 |
    | Helpers | 8331 | 6872 | 1 | 749 | 749 | 7 |
    | Models | 52737 | 41014 | 988 | 4511 | 4 | 7 |
    | Workers | 534 | 451 | 18 | 29 | 1 | 13 |
    | Chanko units | 14617 | 12386 | 5 | 196 | 39 | 61 |
    | Libraries | 40694 | 32471 | 408 | 3238 | 7 | 8 |
    | Feature specs | 24109 | 19915 | 0 | 119 | 0 | 165 |
    | Request specs | 5925 | 5035 | 0 | 11 | 0 | 455 |
    | Routing specs | 418 | 330 | 0 | 1 | 0 | 328 |
    | Controller specs | 39965 | 32978 | 5 | 99 | 19 | 331 |
    | Helper specs | 2183 | 1831 | 0 | 5 | 0 | 364 |
    | Model specs | 42931 | 35051 | 4 | 61 | 15 | 572 |
    | Worker specs | 868 | 717 | 0 | 1 | 0 | 715 |
    | Chanko unit specs | 12616 | 10365 | 0 | 46 | 0 | 223 |
    | Library specs | 16298 | 13515 | 24 | 107 | 4 | 124 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    | Total | 289412 | 234621 | 1710 | 11247 | 6 | 18 |
    +----------------------+-------+-------+---------+---------+-----+-------+
    Code LOC: 234621 Test LOC: 0 Code to Test Ratio: 1:0.0
    Friday, July 26, 13

    View full-size slide

  20. Model classes 861 → 988
    Feature specs LOC 17381 → 19915
    Summary of differences
    Friday, July 26, 13

    View full-size slide

  21. 00:09:27.001 (05) 2085 examples, 0 failures, 6 pendings
    00:09:27.001 (05)
    00:09:27.001 (05) Took 355.818754421 seconds
    00:09:27.098 (05) parallel:spec --> 0.010000 0.090000 1551.140000 (359.309562)
    00:09:27.098 (05) cookpad:spec --> 0.370000 0.190000 1551.600000 (359.778909)
    00:09:27.739 INFO: Total Results --------
    00:09:27.740
    00:09:27.740 Congratulation! All Tests passed.
    00:09:27.740 13177 examples, 0 failure, 162 pending
    00:09:30.608 ** Invoke cookpad:spec:ci:post_growth_forecast (first_time)
    00:09:30.609 ** Execute cookpad:spec:ci:post_growth_forecast
    00:09:30.650
    00:09:30.651 Statistical summary of spec execution time:
    00:09:30.651
    00:09:30.651 total 9528.663149494996
    00:09:30.651 mean 9.144590354601723
    00:09:30.651 variance 410.34485361844173
    00:09:30.651 standard deviation 20.256970494583875
    00:09:30.651
    00:09:30.652 minimum 0.000853474
    00:09:30.652 1st quartile 0.8062394975
    00:09:30.652 median 2.527339377
    00:09:30.652 3rd quartile 8.658470989000001
    00:09:30.652 maximum 198.142290118
    00:09:30.652
    00:09:30.652 Slow Spec Files:
    00:09:30.653
    00:09:30.653 ./spec/features/device/recipe_spec.rb --> 3 min 18.142290118000005 sec
    00:09:30.653 ./spec/features/recipes_hot_spec.rb --> 3 min 9.98139715100001 sec (18
    Friday, July 26, 13

    View full-size slide

  22. 00:09:27.001 (05) 2085 examples, 0 failures, 6 pendings
    00:09:27.001 (05)
    00:09:27.001 (05) Took 355.818754421 seconds
    00:09:27.098 (05) parallel:spec --> 0.010000 0.090000 1551.140000 (359.309562)
    00:09:27.098 (05) cookpad:spec --> 0.370000 0.190000 1551.600000 (359.778909)
    00:09:27.739 INFO: Total Results --------
    00:09:27.740
    00:09:27.740 Congratulation! All Tests passed.
    00:09:27.740 13177 examples, 0 failure, 162 pending
    00:09:30.608 ** Invoke cookpad:spec:ci:post_growth_forecast (first_time)
    00:09:30.609 ** Execute cookpad:spec:ci:post_growth_forecast
    00:09:30.650
    00:09:30.651 Statistical summary of spec execution time:
    00:09:30.651
    00:09:30.651 total 9528.663149494996
    00:09:30.651 mean 9.144590354601723
    00:09:30.651 variance 410.34485361844173
    00:09:30.651 standard deviation 20.256970494583875
    00:09:30.651
    00:09:30.652 minimum 0.000853474
    00:09:30.652 1st quartile 0.8062394975
    00:09:30.652 median 2.527339377
    00:09:30.652 3rd quartile 8.658470989000001
    00:09:30.652 maximum 198.142290118
    00:09:30.652
    00:09:30.652 Slow Spec Files:
    00:09:30.653
    00:09:30.653 ./spec/features/device/recipe_spec.rb --> 3 min 18.142290118000005 sec
    00:09:30.653 ./spec/features/recipes_hot_spec.rb --> 3 min 9.98139715100001 sec (18
    Friday, July 26, 13

    View full-size slide

  23. 00:09:27.001 (05) 2085 examples, 0 failures, 6 pendings
    00:09:27.001 (05)
    00:09:27.001 (05) Took 355.818754421 seconds
    00:09:27.098 (05) parallel:spec --> 0.010000 0.090000 1551.140000 (359.309562)
    00:09:27.098 (05) cookpad:spec --> 0.370000 0.190000 1551.600000 (359.778909)
    00:09:27.739 INFO: Total Results --------
    00:09:27.740
    00:09:27.740 Congratulation! All Tests passed.
    00:09:27.740 13177 examples, 0 failure, 162 pending
    00:09:30.608 ** Invoke cookpad:spec:ci:post_growth_forecast (first_time)
    00:09:30.609 ** Execute cookpad:spec:ci:post_growth_forecast
    00:09:30.650
    00:09:30.651 Statistical summary of spec execution time:
    00:09:30.651
    00:09:30.651 total 9528.663149494996
    00:09:30.651 mean 9.144590354601723
    00:09:30.651 variance 410.34485361844173
    00:09:30.651 standard deviation 20.256970494583875
    00:09:30.651
    00:09:30.652 minimum 0.000853474
    00:09:30.652 1st quartile 0.8062394975
    00:09:30.652 median 2.527339377
    00:09:30.652 3rd quartile 8.658470989000001
    00:09:30.652 maximum 198.142290118
    00:09:30.652
    00:09:30.652 Slow Spec Files:
    00:09:30.653
    00:09:30.653 ./spec/features/device/recipe_spec.rb --> 3 min 18.142290118000005 sec
    00:09:30.653 ./spec/features/recipes_hot_spec.rb --> 3 min 9.98139715100001 sec (18
    Friday, July 26, 13

    View full-size slide

  24. 00:09:27.001 (05) 2085 examples, 0 failures, 6 pendings
    00:09:27.001 (05)
    00:09:27.001 (05) Took 355.818754421 seconds
    00:09:27.098 (05) parallel:spec --> 0.010000 0.090000 1551.140000 (359.309562)
    00:09:27.098 (05) cookpad:spec --> 0.370000 0.190000 1551.600000 (359.778909)
    00:09:27.739 INFO: Total Results --------
    00:09:27.740
    00:09:27.740 Congratulation! All Tests passed.
    00:09:27.740 13177 examples, 0 failure, 162 pending
    00:09:30.608 ** Invoke cookpad:spec:ci:post_growth_forecast (first_time)
    00:09:30.609 ** Execute cookpad:spec:ci:post_growth_forecast
    00:09:30.650
    00:09:30.651 Statistical summary of spec execution time:
    00:09:30.651
    00:09:30.651 total 9528.663149494996
    00:09:30.651 mean 9.144590354601723
    00:09:30.651 variance 410.34485361844173
    00:09:30.651 standard deviation 20.256970494583875
    00:09:30.651
    00:09:30.652 minimum 0.000853474
    00:09:30.652 1st quartile 0.8062394975
    00:09:30.652 median 2.527339377
    00:09:30.652 3rd quartile 8.658470989000001
    00:09:30.652 maximum 198.142290118
    00:09:30.652
    00:09:30.652 Slow Spec Files:
    00:09:30.653
    00:09:30.653 ./spec/features/device/recipe_spec.rb --> 3 min 18.142290118000005 sec
    00:09:30.653 ./spec/features/recipes_hot_spec.rb --> 3 min 9.98139715100001 sec (18
    We use Ruby 2.0.0-p247
    Friday, July 26, 13

    View full-size slide

  25. The problem is still remaining
    1) Tests are continuously growing
    2) The reason of test execution time under 10min
    is that Ruby 2.0 is faster than REE
    Friday, July 26, 13

    View full-size slide

  26. How does it come?
    Friday, July 26, 13

    View full-size slide

  27. One large monolithic Rails application
    Friday, July 26, 13

    View full-size slide

  28. One large monolithic Rails application
    Friday, July 26, 13

    View full-size slide

  29. One large monolithic Rails application
    Multiple Rails applications
    sharing models via API
    Friday, July 26, 13

    View full-size slide

  30. Friday, July 26, 13

    View full-size slide

  31. Friday, July 26, 13

    View full-size slide

  32. Applications will be increased,
    but they are too small than cookpad.
    Friday, July 26, 13

    View full-size slide

  33. Applications will be increased,
    but they are too small than cookpad.
    Test execution time won’t be raised
    if their CI jobs are executed in parallel.
    Friday, July 26, 13

    View full-size slide

  34. How to manage CI?
    Friday, July 26, 13

    View full-size slide

  35. Current CI
    Friday, July 26, 13

    View full-size slide

  36. Friday, July 26, 13

    View full-size slide

  37. ci-slave-ruby200-001 ci-slave-ruby200-002
    ci-slave-ruby200-00{3,4,5} ci-slave-ruby200-00{2,3}-midnight
    mysql56 qt48
    mysql55 qt47
    cookpad
    mysql55 qt47
    mysql55 qt47
    elasticsearch ruby200
    m1.large
    ruby200
    ruby200 ruby193
    ruby200 ruby193
    midnight
    Node labels
    Friday, July 26, 13

    View full-size slide

  38. ci-slave-...-001
    ci-slave-...-002
    ci-slave-...-003
    ci-slave-...-004
    ci-slave-...-005
    ci-slave-...-002-midnight
    ci-slave-...-003-midnight
    8am 10pm
    Nodes’ availability
    Friday, July 26, 13

    View full-size slide

  39. ci-slave-ruby200-001 ci-slave-ruby200-002
    ci-slave-ruby200-00{3,4,5} ci-slave-ruby200-00{2,3}-midnight
    mysql56 qt48
    mysql55 qt47
    cookpad
    mysql55 qt47
    mysql55 qt47
    elasticsearch ruby200
    m1.large
    ruby200
    ruby200 ruby193
    ruby200 ruby193
    midnight
    Node labels
    Friday, July 26, 13

    View full-size slide

  40. Developers DevInfra Omukins
    cookpad
    cafe
    omuken
    Friday, July 26, 13

    View full-size slide

  41. Friday, July 26, 13

    View full-size slide

  42. Demonstration
    Friday, July 26, 13

    View full-size slide

  43. github.com/mrkn/whitesnake
    Fork
    m
    e
    on
    GitHub
    Friday, July 26, 13

    View full-size slide

  44. !e End
    Friday, July 26, 13

    View full-size slide

  45. bit.ly/cookpad_jobs
    Friday, July 26, 13

    View full-size slide