From 'Legacy' to 'Edge' 2014 Edition

Transcript

1. !
   From ‘Legacy' to ‘Edge'
   2014 edition
   !
   Hiroshi SHIBATA
   @hsbt
   RubyConfTaiwan 2014
   

   View Slide

2. Thank you!
   At first, Thank you so much for accept to my proposal. I love taiwan
   and taipei.
   

   View Slide

3. SHIBATA Hiroshi(@hsbt)
   My name is Hiroshi SHIBATA, Please call me hiroshi or shibata. my
   twitter and github handle name is hsbt.
   

   View Slide

4. Ruby Committer
   I’m ruby committer. I prevent that ruby break rails and your
   application. sometimes I fix ruby, fix rails, fix my code.
   

   View Slide

5. ruby-lang.org
   And I’m maintaining ruby-lang.org. I support ruby commiter and make
   easy to get rubies all over the world like CDN.
   !
   Our site is hosted by github and heroku. If you find something to
   improve and documentation errors in www.ruby-lang.org, you can fix
   and send a pull request. we can merge it.
   

   View Slide

6. Rails Girls in Japan
   I’m one of organizers in RailsGirls in Japan.
   

   View Slide

7. asakusa.rb
   And I’m member of asakusa.rb. asakusa.rb is the most active meetup
   group in japan. we are talking about ruby and writing code every
   Tuesday.
   

   View Slide

8. I’m working for GMO pepabo in japan. we provide server hosting, blog
   service, shopping mall service etc. we have a lot of web services.
   

   View Slide

9. ϛυϧ΢ΣΞͷΞοϓάϨʔυ
   ࣾ಺։ൃج൫ͷߏங
   ౷ܭج൫ͷߏங
   +409ͷӡ༻੔උ
   HJUIVC
   ηΩϡϦςΟ؂ࠪ
   ίʔυϨϏϡʔ
   ςετج൫ͷߏங
   ΞΫηεղੳج൫ͷߏங
   ։ൃϓϩηε
   ৽ਓڭҭ
   ݪߘࣥච
   ΧϯϑΝϨϯεൃද
   044։ൃ αʔόʔߏ੒؅ཧͷ࡮৽
   ϦʔϯελʔτΞοϓ
   +FOLJOT
   3VCZ3BJMT
   3%#.4
   ٕज़ج൫νʔϜ
   I’m at technical platform team on company. example for framework
   upgrade and opensource development, building to analysis platform
   etc.
   

   View Slide

10. http://30d.jp
    We are running photo sharing service named 30days album. Service
    concept of the service is private photo sharing.
    

    View Slide

11. from 2008/4
    30days launched in April, two thousand eight.
    

    View Slide

12. 380,000 users
    230,000,000 photos
    we have three hundred eighteen thousand users, and two hundred
    thirteen million photos.
    

    View Slide

13. Our Rails app
    46 models
    5000 lines in controllers
    400 lines in routes.rb
    1:1.3 code to test ratio
    our rails is traditional rails. probably our rails is same as your rails.
    

    View Slide

14. system architecture
    but our system architecture is too complex system. I’m going to
    describe system architecture of 30days.
    

    View Slide

15. application server
    these are application servers. we separate to customer transaction
    database from application database.
    

    View Slide

16. storage server
    these are storage servers. we don’t use cloud platform. these are build
    on on-premises platform.
    

    View Slide

17. job server
    job servers handles to image manipulation workers. we have a lot of
    job pattern like rotate and resize.
    

    View Slide

18. transaction server
    we can access only office network to customer transaction server.
    

    View Slide

19. storage is over
    450 TB
    our storage has over four hundred fifteen tera bytes of photos.
    

    View Slide

20. database size
    is over
    250 GB
    our database has over 250GB of data. It use MySQL 5.5 with barracuda
    plugin.
    

    View Slide

21. Development
    Culture
    I’ll introduce the our team background and culture.
    

    View Slide

22. github workflow
    We use workflow like a github’s one. master branch is always to allow
    deployment and all-green test states. all of code is reviewed by others.
    

    View Slide

23. IRC
    github issue
    costomer’s
    contact
    nagios
    we are using IRC(Internet relay chat). we integrate all of alert and
    monitoring messages to it.
    

    View Slide

24. DevOps
    we are using modern operation tools. puppet and newrelic, munin,
    nagios and more.
    

    View Slide

25. 4 deploy/day
    we are deploying one to four times per everyday.
    

    View Slide

26. I delete unused code. this figure is our team graph of github. deleting
    code is bigger than writing code. my main work is code vanishment.
    

    View Slide

27. How to migrate
    edge rails
    from
    legacy rails?
    It’s Today’s main theme, I’ll introduce the migration strategy.
    

    View Slide

28. Ruby 1.8.6
    Rails 2.0.2
    30days had been running under ruby 1.8.6 and rails 2.0.2 when I
    joined. I was very surprised. because latest rails was 3.2 at that time.
    

    View Slide

29. Ruby 1.8.6
    Rails 2.0.2
    Ruby 2.1.1
    Rails 4.1.0 new!
    I decided to upgrade ruby and rails. and started thinking aboud
    migration strategy.
    

    View Slide

30. Why?
    Why we need to migrate and upgrade Ruby and Rails?
    

    View Slide

31. the answer is this tweet. he said “Ruby 2.1.1 is too fast. We can make
    faster our web application only upgrading language version, It seems
    joke to me.”
    

    View Slide

32. Ruby on Rails
    It’s true. Rails is Ruby code. We need to focus to Ruby.
    

    View Slide

33. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    this matrix shows compatibility between ruby and rails.
    

    View Slide

34. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    Slow
    Fast
    In Generally, Upgrading Ruby give more faster experience for us, but
    Upgrading Rails give more slower experience for us.
    

    View Slide

35. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    Red line shows big wall of Upgrading Rails, and Blue line shows big
    wall of Upgrading Ruby for us
    

    View Slide

36. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    Our strategy is red arrows routes.
    

    View Slide

37. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    At first we jumped Rails 2.3 and Ruby 1.8.7.
    

    View Slide

38. Rails 2.0
    Rails 2.3
    I’ll describing our branch strategy. we created Rails 2.3 branch such as
    above figure. master branch is rails 2.0 line.
    

    View Slide

39. Rails 2.0
    Rails 2.3
    hotfix A
    migration A
    we made hot fix branch from Rails 2.0 branch. and made a migration branch from
    Rails 2.3, migration branch contains to rewrite code adopting more rails way, etc.
    

    View Slide

40. Rails 2.0
    Rails 2.3
    hotfix A
    migration A
    hotfix branch is finished, we merge it to master branch and backport
    to rails 2.3 branch.
    

    View Slide

41. Rails 2.0
    Rails 2.3
    hotfix A
    migration A
    we finished migration branch, I merged it to rails 2.3 branch. I merged
    whole hotfixes and migrations to rails 2.3 branch.
    

    View Slide

42. It’s real branch status like rainbow line.
    

    View Slide

43. Zero
    Downtime
    Deploy
    one of the most offensive operation is “zero downtime deploy”
    

    View Slide

44. cookpad/kage
    well, I’ll introduce kage.
    

    View Slide

45. Kage (kah-geh) is an HTTP shadow
    proxy server that sits between
    clients and your server(s) to
    enable "shadow requests".
    kage is an http shadow proxy server that sits between clients and your server to
    enable “shadow requests”. kage is developed by cookpad. cookpad is best recipe
    sharing service in japan.
    

    View Slide

46. the basic of kage
    real request
    this figure is basic usage of kage. we put kage at the back of nginx. while kage
    proxies a request to production, it duplicate the request and pass though it to
    staging environment.
    we can test big changes with real request now. kage can build powerful test suites.
    

    View Slide

47. but sometimes kage caused trouble to our system.our service can receive big
    pictures. perhaps it caused this problem I guess ;)
    

    View Slide

48. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    Next plan is very hard time for us, Upgrading Rails 3
    

    View Slide

49. rails 2.3 with bundler
    I finished to integrate to rails 2.3 and bundler. we can use a lot of
    gems. example for new relic, delayed_job, rack, and more
    

    View Slide

50. source 'https://rubygems.org'!
    !
    gem 'rails', '~> 2.3.18'!
    gem 'rake', '~> 0.9.2'!
    gem 'rdoc'!
    gem 'rake-confirm'!
    !
    gem 'mysql'!
    gem "mysql_retry_lost_connection"!
    gem 'acts_as_paranoid', :github => 'paperboy-30days/acts_as_paranoid'!
    gem 'passenger', '~> 3.0'!
    gem 'memcache-client', :require => 'memcache'!
    gem 'system_timer'!
    gem 'yajl-ruby', :require => 'yajl'!
    gem 'will_paginate', '~> 2.3'!
    gem 'mail'!
    !
    gem 'sass'!
    gem 'compass-rails'!
    !
    gem 'osaipo_client', :git => 'git@github.com:paperboy-all/
    osaipo_client.git', :branch => 'legacy'!
    gem 'jugem_client', :git => 'git@github.com:paperboy-all/
    jugem_client.git'!
    gem 'ppb_footer', :github => 'paperboy-all/ppb_footer'!
    We extracted some plugins, acts_as_paranoid was put vendor/plugin folder, now
    we moved on our github organization. and we moved our external service client
    in library folder to organization folder.
    

    View Slide

51. require "./config/environment"!
    !
    use Rails::Rack::LogTailer!
    use Rails::Rack::Static!
    run ActionController::Dispatcher.new
    rack with rails 2.3
    we are using rack with rails 2.3, it can be easily applied to rails 2.3.
    Because we need to migrate rack in the future.
    

    View Slide

52. 30days-front
    can’t use
    hyphen
    can’t use
    number
    can’t convert
    !
    application.rb...
    but, I have one of big problem. our repository name can’t be converted to rails 3
    style namespace. because our repository use hyphen and integer.
    

    View Slide

53. Reduce
    customer frustration.
    I did works only upgrading. We need to write a new code for customer
    requirements and business goal.
    

    View Slide

54. app1
    app2
    backgroundrb to dj
    I replaced old job worker named backgroundrb. backgroundrb is on-memory system,
    so it’s cleared when memory refresh. It means all job is deleted. We and our user got
    frustration on this situation. I replaced all of job system to persistent workers like
    delayed_job.
    

    View Slide

55. this is Pull Request of Upgrading branch. commits are over than two
    hundred commits. file changed are over the three hundred sixteen five.
    

    View Slide

56. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    After Upgrading Rails 3, I started to upgrade Ruby runtime.
    

    View Slide

57. Use trunk everyday.
    Basically, I’m Ruby commiter. I use trunk and head of all branches
    builds.
    

    View Slide

58. It’s easy to find to defects Ruby, Rubygems, bundler. I report our
    tracking system. and fixed it quickly.
    

    View Slide

59. this is my rails fixes. I aimed to targeted version of Ruby is 2.1. I fixed
    all of interruptions for our Application with Ruby 2.1.
    

    View Slide

60. 1.8 to 2.0
    How to upgrade Ruby 1.8 to 2.0.
    

    View Slide

61. my friends named mrkn. he talked that how to upgrade Ruby 1.9.3. In
    this talk contains a lot of technics in the wild. Please read it later.
    

    View Slide

62. 2.0 to 2.1
    2.0 to 2.1 has no trouble. it cares backward compatible.
    

    View Slide

63. CPU Usage(%)
    0
    25
    50
    75
    100
    49
    69
    2.0 2.1
    This is benchmark for new relic average score. CPU Usage is reduced
    twenty percents on Ruby 2.1
    

    View Slide

64. Resp. time(ms)
    0
    50
    100
    150
    200
    250
    189
    221
    2.0 2.1
    It’s Response Time comparison. Ruby 2.1 reduces twenty to thirty
    percents of response time.
    

    View Slide

65. memory usage(giga byte)
    0
    1
    2
    3
    4
    3.8
    3.2
    2.0 2.1
    But memory usage are increased 10-20% by Ruby 2.1. If you use PaaS
    service like heroku, you need to evaluate this situation before
    upgrading Ruby 2.1
    

    View Slide

66. https://www.ruby-lang.org/en/news/2014/03/10/regression-of-hash-reject-in-ruby-2-1-1/
    I noticed you, Ruby 2.1.1 have regression feature. It’s difference behaviour of reject
    method on Inheritance Hash. Please read our announcement. This regression is
    already fixed Rails 4.0 and 4.1, If you use Rails 3.2, You need to applied workaround
    patch.
    

    View Slide

67. Ruby
    1.8.6 1.8.7 1.9.3 2.0.0 2.1
    Rails
    2.0/2.1
    2.3
    3.0
    3.2
    4.0
    After Upgrade Ruby 2.1, I started Upgrading Rails 4.0 line.
    

    View Slide

68. Details of Upgrading Rails 4.0, they are written by this book. this book
    is free e-book. I recommended this book.
    

    View Slide

69. Use modern architecture
    I replaced modern architecher to upgrade Rails 4.
    

    View Slide

70. app1
    app2
    Our System used Kyoto Tycoon. Kyoto Tycoon is memcached compatible session
    storage, it have dual master function. but session client named dalli not support
    Kyoto tycoon on Rails 4.
    

    View Slide

71. kyoto tycoon is slower
    than mysql…
    And biggest problem is this. Kyoto Tycoon is slower than mysql. It’s
    caused by dual master mechanism
    

    View Slide

72. I replaced pure memcached. I solved all problem for upgrading Rails 4
    

    View Slide

73. Resp. time(ms)
    0
    60
    120
    180
    240
    300
    287
    189
    3.2 4.0
    This is benchmark of Rails4. rails 4 is slower than rails 3. it increased
    100 micro seconds of response time.
    

    View Slide

74. Ruby
    1.9.3 2.0.0 2.1 2.2
    Rails
    3.0
    3.2 ?
    4.0 ?
    4.1 ?
    this is upgraded version of Ruby and Rails combination matrix. I
    dropped Ruby 1.8 and Rails 2.3.
    

    View Slide

75. Ruby
    1.9.3 2.0.0 2.1 2.2
    Rails
    3.0
    3.2 ?
    4.0 ?
    4.1 ?
    New!!
    I upgraded Rails 4.1 last week!
    

    View Slide

76. I recommend to official upgrade guides
    

    View Slide

77. 1. Flash structure changes
    2. Mutator methods called on Relation
    And I notice big incompatible things on Rails 4.1. One Flash structure changes. Rails changed key
    class of flash to String from Symbol. Two. Mutator methods called on Relation. before Rails 4.1
    allows mutator methods like map exclamation on Relation objects. but Rails 4.1 raised this
    methods.
    

    View Slide

78. And changed namespace of Minitest. If you use ActionSupport::Testcase and test
    helper libraries like webmock. Probably Minitest broke these helpers. this is my Pull
    request of web mock, I already fixed it.
    

    View Slide

79. Resp. time(ms)
    0
    60
    120
    180
    240
    300
    266
    287
    4.0 4.1
    It’s benchmark of Rails 4.0 and 4.1. Rails 4.1 is faster than 4.0. but it’s
    slow than Rails 3.0 yet.
    

    View Slide

80. Ruby
    1.9.3 2.0.0 2.1 2.2
    Rails
    3.0
    3.2 ?
    4.0 ?
    4.1 ?
    I started to evaluate Rails 4.1 with Ruby 2.2. Ruby 2.2 will be releasing
    December 2014 for us. I work hard to solve incompatible changes
    before releases.
    

    View Slide

81. class Bar!
    def bar(foo = foo())!
    foo!
    end!
    !
    def buzz(foo = foo)!
    foo!
    end!
    !
    def foo!
    :buzz!
    end!
    end!
    !
    p Bar.new.bar!
    p Bar.new.buzz
    This is example of incompatible changes on Ruby 2.2. class Bar have
    foo equal foo method arguments. bar method called foo methods,
    buzz method doesn’t called foo methods.
    

    View Slide

82. % ruby -v r45272.rb!
    ruby 2.2.0dev (2014-04-13 trunk 45580) [x86_64-darwin13]!
    :buzz!
    nil!
    !
    % ruby -v r45272.rb!
    ruby 2.1.2p80 (2014-03-01 revision 45231) [x86_64-darwin13.0]!
    :buzz!
    :buzz
    ruby 2.2. returns nil. it isn’t assignment same name of default value.
    

    View Slide

83. This changes break Rails. but it’s already fixed Rails 4.1 by Aaron
    Patterson. this patch is backported Rails 4.0. but didn’t fix 3.2 branch.
    If you still use Rails 3.2, You can’t use Ruby 2.2.
    

    View Slide

84. conclusion
    Conclusion.
    

    View Slide

85. 2. fix gem
    4. fix Ruby
    3. fix Rails
    1. fix your code
    Fix your code. fix using gems. fix Rails. fix Ruby. You should get more
    controllable things. It’s expand your life!
    

    View Slide