Slide 1

Slide 1 text

ࣲాതࢤ SHIBATA Hiroshi QBQFSCPZDP BTBLVTBSC paperboy&co., Inc. Ruby ͷෆ۩߹ͷݟ͚ͭํ ൃද৔ॴRubyConf Taiwan 2012 2012-12-8(Sat) How to discover the Ruby's defects with web application.

Slide 2

Slide 2 text

㟬޷ ni-hao.

Slide 3

Slide 3 text

จࣈͷς ΩετΒ͠ ͍Ͱ͢Αʁ େৎ෉͔ʁ Xie Xie rubytaiwan. Lai tai wan, wan hen kai xin.

Slide 4

Slide 4 text

SHIBATA Hiroshi(@hsbt) Wo shi cong ri ben lai de. Wo jiao Hiroshi, Shibata. My name is Hiroshi SHIBATA, please call me Hiroshi or SHIBATA. This jacket is my trademark and fighting armor. Today is same jacket.

Slide 5

Slide 5 text

Thanks for @ihower and @rubytaiwan accepting my proposal. I appreciate it. I’m super exciting. Taiwan is good city. I love taiwan. I decided to arrived rubyconf taiwan in next year.

Slide 6

Slide 6 text

asakusa.rb I’m member of asakusa.rb. asakusa.rb is inspired by Seattle.rb. This meetup start every tuesday. this meetup founder is Akira Matsuda. If you arrived tokyo, japan, please join our meetup.

Slide 7

Slide 7 text

I’m working at paperboy&co. My job is solving for all technical problems such as continuous deployment, security issue and gaining code quality etc... it’s company wide scrum master.

Slide 8

Slide 8 text

http://fanic.jp Our one of the rails service is fanic. fanic is that you can have personal music store. I’m sorry, fanic is only japanese language available.

Slide 9

Slide 9 text

http://musiccaptains.com/zh/2012/11/27/fanic- music-website-song-purchase/ You are lucky. Some of chinese writer wrote introduction of fanic. this entry is chinese! Please join it.

Slide 10

Slide 10 text

tDiary In this talk, I’ll introduce tDiary and how tDiary can discover ruby’s defects. I don’t talk agile and how to maintain legacy code. If you interested these theme, please talk to me after session.

Slide 11

Slide 11 text

http://github.com/tdiary tDiary is hosted by github now, It this url. I’m sorry this site is only available for japanese and english information. if you interested tDiary, please join our organization.

Slide 12

Slide 12 text

͆hat’s? First, what’s tDiary? Do you know tDiary? Please raise up your hand?

Slide 13

Slide 13 text

Nikki System tDiary is Nikki System. Nikki is Diary. Japanese internet has web nikki culture before blog.

Slide 14

Slide 14 text

Like a Blog Nikki is a blog in 2012. We don’t divide their.

Slide 15

Slide 15 text

Specification Overview I’ll introduce tDiary mainly specification.

Slide 16

Slide 16 text

3. Pluggable 2. CGI/Rack 1. Ruby First, it use Ruby, and this is most important things. Second, running environments are traditional CGI and Rack. It means tDiary can run in all of application server. For example apache, passenger and unicorn. Third, it has pluggable mechanism. It mechanism is possibled by ruby meta-programming. this feature supports newbie rubyists.

Slide 17

Slide 17 text

History Second, I talk about history.

Slide 18

Slide 18 text

Ruby 1.6.7 2002 In 2002, tDiary began to development in the sf.net and used CVS. I joined tDiary developper team this year. First version supported Ruby-1.6.7. Please raise up your hands if you had used this version of Ruby?

Slide 19

Slide 19 text

Ruby 1.8.0 2003 In 2003, tDiary supported Ruby-1.8.0. This version included many standard libraries.

Slide 20

Slide 20 text

What is Ruby needed to run the tDiary? “What is Ruby needed to run the tDiary.” This phrase posted some contents in 2004. It means tDiary more famous lather than Ruby. Everyone knows tDiary in Japan.

Slide 21

Slide 21 text

Ruby 1.9.0 2008 In 2008, tDiary supported 1.9.0. Big internal changing occurred in this version. All of String have Encoding. tDiary was broken by encoding changes. It was so hard for us, and we have be supporting now.

Slide 22

Slide 22 text

2013 In 2013. it’s future.

Slide 23

Slide 23 text

Ruby 2.0.0 Probably, tDiary is supporting 2.0. This feature is assigned me.

Slide 24

Slide 24 text

As you can see that tDiary is growing up with Ruby. In other hand, Ruby is growing up with tDiary.

Slide 25

Slide 25 text

tDiary discovered Ruby’s defects In other words, tDiary discovered some of ruby’s defects.

Slide 26

Slide 26 text

why? Why tDiary be able to discover Ruby’s defects?

Slide 27

Slide 27 text

Background I’m going to describe Japanese Internet environment. Nearly 2004, web programmer need to use shared hosting server for web programming.

Slide 28

Slide 28 text

tDiary can use plugin written by tiny ruby code. Implementation of this feature need to ruby-meta-programming.

Slide 29

Slide 29 text

eval tDiary’s core feature is using eval, instance_eval and class_eval.

Slide 30

Slide 30 text

private # loading tdiary.conf in current directory def configure_attrs @secure = true unless @secure @options = {} eval( File::open( 'tdiary.conf' ) {|f| f.read }.untaint, b, "(tdiary.conf)", 1 ) # language setup @lang = 'ja' unless @lang begin This is part of tDiary’s code. tDiary configuration was written by Ruby code. it’s not by yaml and json.

Slide 31

Slide 31 text

# The value must be the Array of Ruby and Array's contents are String # of Ruby. Though these Strings are converted to Regular Expression # of Ruby when compared, you can't use Regexp of Ruby. @no_referer = [ '^' + Regexp.quote( base_url ), # Your diary '^http://localhost[:/]', '^http://192.168.', '^http://172.1[6789]', '^http://172.2[0-9]', '^http://172.3[01]', '^http://10.', ] # URLs which are only recorded into Today's Link (Regular Expression) # @only_volatile is an array of URLs to record into only Today's Link. # When a referer match to this list, it will be recoreded to volatile # list. This list will be cleared when you make new text in new day. # Specify @only_volatile same style of @no_referer. @only_volatile = [ ] # The rules which convert the specified URL to the word (Regular Expression) # @referer_table is configured so that readable URLs are shown in This is example of tdiary configuration file. If you try to customize tDiary behavior, you should write method definitions, variable settings and more into “tdiary.conf”. It’s very useful.

Slide 32

Slide 32 text

private def setup_attr_accessor_to_all_ivars names = instance_variables().collect {|ivar| ivar.to_s.sub(/ @/, '') } (class << self; self; end).class_eval { attr_accessor *names } end def configure_bot_pattern bot = ["bot", "spider", "antenna", "crawler", "moget", "slurp"] bot += @options['bot'] || [] @bot = Regexp::new( "(#{bot.uniq.join( '|' )})", true ) end Next point is class_eval. tDiary uses classic “cgi.rb” and creates CGI instance per all requests. CGI instance have a configuration instance. It needs access mechanism for tDiary’s core and plugins. class_eval make configuration instance possible to access it.

Slide 33

Slide 33 text

def load_plugin( file ) @resource_loaded = false begin res_file = File::dirname( file ) + "/#{@conf.lang}/" + File::basename( file ) open( res_file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/ #{@conf.lang}/#{File::basename( res_file )})", 1 ) end @resource_loaded = true rescue IOError, Errno::ENOENT end File::open( file.untaint ) do |src| instance_eval( src.read.untaint, "(plugin/ #{File::basename( file )})", 1 ) end end The i18n of tDiary use Ruby code. this is same the case of configuration files. tDiary doesn’t use i18n gem! In past, tDiary include chinese language. but we dropped it, because we have no maintainer.

Slide 34

Slide 34 text

One of 2.0.0 feature is “Refinements”. tDiary have many monkey patches for Ruby’s core feature. I’m going to fix these patches by using “Refinements”.

Slide 35

Slide 35 text

“eval” is powerful metaprogramming in ruby. but you worry about security issue.support two things. one, share hosting environment. imagine of you made cgi in shared hosting environment. that cgi can use code written by other people. If this code is exploit code. How do you protect it?

Slide 36

Slide 36 text

$SAFE tDiary protects with variable of $SAFE against security issue. I’m introducing $SAFE mechanism in Ruby. Did you know this feature?

Slide 37

Slide 37 text

open( res_file.untaint ) do | src| instance_eval( src.read.untaint, "(plugin/#{@conf.lang}/ #{File::basename( res_file )})", 1 ) end @resource_loaded = true rescue IOError, Errno::ENOENT end File::open( file.untaint ) do | src| instance_eval( src.read.untaint, "(plugin/ #{File::basename( file )})", 1 ) end end I talk overview of $SAFE mechanism. $SAFE mechanism makes all object taint. Once you use $SAFE, you can’t use tainted object in some function. If you want to use tainted objects, you should call “untaint” method.

Slide 38

Slide 38 text

$SAFE=1 $ ruby -e '$SAFE = 1; open(ARGV[0])' foo -e:1:in `initialize': Insecure operation - initialize (SecurityError) from -e:1 $SAFE level 1 disallow “File.open” with tainted strings. If you want to open with ARGV[0] string. you should call “untaint”.

Slide 39

Slide 39 text

$SAFE=4 % ruby -e '$SAFE=4; eval("p :foo".untaint)' -e:1:in `untaint': Insecure operation at level 4 (SecurityError) from -e:1:in `' % ruby -e '$SAFE=1; eval("p :foo".untaint)' :foo $SAFE level 4. If you use untainted strings with eval, this level disallow using it. tDiary customization is handled with eval and $SAFE mechanism.

Slide 40

Slide 40 text

=begin == Safe module =end module Safe def safe( level = 4 ) result = nil if $SAFE < level then Proc.new { $SAFE = level result = yield }.call else result = yield end result end module_function :safe end It’s protips. If you use different $SAFE level in same code. you need to use “Proc#new” and “Proc#call”. tDiary use this procedure when running with shared environment.

Slide 41

Slide 41 text

bugreport http://bugs.ruby-lang.org/issues/5279 fixed r33328: * encoding.c (require_enc): reject only loading from untrusted load paths. [ruby-dev:44541] [Bug #5279] * transcode.c (load_transcoder_entry): ditto. assume system default tmpdir safe. [ruby-dev:42089] http://bugs.ruby-lang.org/issues/3733 fixed r29209: assume system default tmpdir safe. [ruby-dev:42089] http://bugs.ruby-lang.org/issues/1115 * load.c (rb_require_safe): raises when the path to be loaded is tainted. [ruby-dev:37843] I've never seen that other people are using $SAFE feature. therefore $SAFE feature is very buggy.

Slide 42

Slide 42 text

http://bugs.ruby-lang.org/issues/1115 * load.c (rb_require_safe): raises when the path to be loaded is tainted. [ruby-dev:37843] % ruby -e '$SAFE=1;p require "zlib"' -e:1:in `require': Insecure operation - require (SecurityError) from -e:1:in `' % ruby -e '$SAFE=1;p require "English"' -e:1:in `require': Insecure operation - require (SecurityError) from -e:1:in `' This defect is that ruby raises exception when ruby call require with tainted path.

Slide 43

Slide 43 text

http://bugs.ruby-lang.org/issues/3733 fixed r29209: assume system default tmpdir safe. [ruby-dev:42089] % ruby -e '$SAFE=1;File.expand_path(".".taint)' -e:1:in `expand_path': Insecure operation - expand_path (SecurityError) from -e:1:in `' This defects. if you use “tmpdir” running with $SAFE level 4, Ruby raised exception.

Slide 44

Slide 44 text

http://bugs.ruby-lang.org/issues/5279 fixed r33328: * encoding.c (require_enc): reject only loading from untrusted load paths. [ruby-dev:44541] [Bug #5279] * transcode.c (load_transcoder_entry): ditto. assume system default tmpdir safe. [ruby-dev:42089] % ruby -e ‘$SAFE = 3; "a".encode("UTF-16")’ -e:1:in `encode': Insecure operation - encode (SecurityError) from -e:1:in `' In this report. If you run some code with $SAFE level 4, and not with calling “encode”. Ruby is raised exception. I already talked about all of String has Encoding in Ruby 1.9. This defects is important, because anyone never use String#encode with $SAFE level 4 in Ruby 1.9

Slide 45

Slide 45 text

จࣈͷς ΩετΒ͠ ͍Ͱ͢Αʁ େৎ෉͔ʁ I’m sorry, I introduce bad news. JRuby isn’t support $SAFE yet.

Slide 46

Slide 46 text

@shugomaeda 4"'&VTFSTJTPOMZ U%JBSZ*XBOUUPESPQ JUJOOFYUWFSTJPOPG SVCZ SFBMMZ @hsbt I think probably dropped out $SAFE feature in 2.1. We accept it. so tDiary need to change with rubies. The Internet environment is changing now. people doesn’t use shared hosting server. they use PaaS like a sqale and heroku.

Slide 47

Slide 47 text

tDiary discover ruby’s defects is easy. In this case is only tDiary being? I think no.

Slide 48

Slide 48 text

tDiary uses Travis-CI. I build ruby-trunk every morning, and run it with tDiary. So, sometimes I get defects in ruby-trunk.

Slide 49

Slide 49 text

@tenderlove said that Rails-3.2 works with Ruby-2.0. It means that many web applications can work with Ruby-2.0. You can begin to use Ruby-2.0 now.

Slide 50

Slide 50 text

Even though, you couldn’t run code with Ruby-2.0 in production, you can run code in test environment. I think modern code have a lot of test code. you can.

Slide 51

Slide 51 text

https://speakerdeck.com/kakutani/above-all-make-it-fun If it was also difficult to run your code in test environment, your hobby code should run with Ruby-2.0. I think that programmer have hobby code. tDiary is hobby code. Matz said “Ruby is hobby code!” too.

Slide 52

Slide 52 text

FFFFFFFFFFFFFFFFFFF.....F FFFFFFFFFFFFF....F.F...FFFF FFFFFFFFFFFFFFFFFFFFFF. ....FFFFFFFFFFFFFF....F.F...F FFFFFFFFFFFFFFFFFFFFFF FFF.....FFFFFFFFFFFFFF....F. F...FFFFFFF.......FFFFFFFFFF When you run hobby code with Ruby-2.0, you will get error or segmentation fault. You should report it ruby core team.

Slide 53

Slide 53 text

bugreport http://bugs.ruby-lang.org/issues/6781 * lib/open-uri.rb: use respond_to? to test Tempfile. [ruby-dev:45995] [Bug #6781] reported by hsbt (Hiroshi SHIBATA). http://bugs.ruby-lang.org/issues/5952 * io.c (argf_next_argv): reset ARGF.next_p on ARGV.replace. r34409 breaks replacing ARGV. [ruby-dev:45160] [Bug #5952] http://bugs.ruby-lang.org/issues/7040 * ext/zlib/zlib.c (zstream_run_func): don't call inflate() when z->stream.avail_in == 0. it return Z_BUF_ERROR. but deflate() could be called with z->stream->avail_in == 0 because it has hidden buffer in z->stream->state (opaque structure). fix for gem install error. [ruby-dev:46149] [Bug #7040] I discovered some defects in ruby-trunk. One of these defects is ruby-trunk can’t install some gems like libv8. Today’s ruby is fixed by my reports.

Slide 54

Slide 54 text

A few weeks ago. I discovered a new feature in cgi.rb. I thought this feature broke backward compatibility. I reported it for ruby-core. In this result, this feature reverted. You need to run your code with early version of Ruby. and report defects, behavior change and new feature. If you detect behavior change after code freeze, it’s too late.

Slide 55

Slide 55 text

Report it! You shouldn’t write these report in blog or twitter. Probably, ruby-core commiters don’t read your tweets.

Slide 56

Slide 56 text

Are commiter only people who develop ruby? I don’t think so. this picture is one of scene in rubyconf. matz, ko1 and JRuby guys and many commiter and people discussion “refinements” Your report and proposal of new feature are making ruby!

Slide 57

Slide 57 text

Run your code with 2.0.0 Let’s run your code with ruby 2.0.0

Slide 58

Slide 58 text

Thanks.