analyze_logs_by_rake

ログ解析にRakeを使うことをオススメします
M3 Tech Talk #18 2014/01/31 #m3dev

Transcript

1. Rake Ͱϩάղੳ M3 Teck Talk #18 2014.01.31 தଜوࢤ
2. Rakeͱ͸ • make ͷruby൛ • JavaͰ͍͏ͱ͜Ζͷant • Rakefileͱ͍͏ϑΝΠϧ໊ͰλεΫΛఆٛ • rakeίϚϯυͰλεΫΛ࣮ߦ
3. rakeͷ֓ཁ • ʮRakefileʯͱ͍͏ϑΝΠϧ໊ͰॲཧΛهड़ • هࡌ͸׬શͳRuby • taskϝιουͰλεΫΛఆٛ • ґଘؔ܎Λఆٛͯ͠Ұ࿈ͷॲཧͷྲྀΕΛ࡞ΕΔ • fileϝιουͰϑΝΠϧΛ࡞੒͢ΔλεΫΛఆٛ • ϑΝΠϧ͕͋ͬͨΓɺґଘ͢ΔϑΝΠϧΑΓ৽͚͠Ε͹εΩοϓ ͢Δ • shϝιουͰγΣϧίϚϯυΛ࣮ߦͰ͖Δ
4. ؆୯ͳྫ͔Β
5. $ rake -P #λεΫҰཡ rake hello rake world hello ! $ rake hello # hello Λ࣮ߦ Hello ! $ rake world # worldΛ࣮ߦ Hello World ! $ rake file1.txt ˌfile1.txtΛ࡞੒ $ ls file1.txt ! $ cat file1.txt 11111 # Rakefile ! task :hello do puts "Hello" end ! task :world => [:hello] do puts "World" end ! file "file1.txt" do |t| sh "echo 1111 > #{t.name}" end
6. $ rake all.txt << create file1.txt >> complete file1.txt << create file2.txt echo 2222 > file2.txt >> complete file2.txt << create all.txt cat file1.txt file2.txt > all.txt >> complete all.txt ! $ cat file1.txt 1111 ! $ cat file2.txt 2222 ! $ cat all.txt 1111 2222 file "file1.txt" do |t| puts "<< create file1.txt" File.open(t.name, "w") do |file| file.puts("1111") end puts ">> complete file1.txt" end ! file "file2.txt" do |t| puts "<< create file2.txt" sh "echo 2222 > #{t.name}" puts ">> complete file2.txt" end ! file "all.txt" => ["file1.txt", "file2.txt"] do |t| ! puts "<< create all.txt" sh "cat #{t.prerequisites.join(' ')} 
 > #{t.name}" puts ">> complete all.txt" end
7. RakeΛ࢖͏͓͢͢ΊϙΠϯτ
8. ͓͢͢ΊϙΠϯτ • RubyͰॻ͚Δ • ̍ͭͷλεΫΛখ͍͞ػೳʹɺґଘؔ܎ͰҰ࿈ͷྲྀΕΛ ࡞Δͱɺػೳ͝ͱʹ࣮ߦͰ͖ͯσόοά͠΍͍͢ • ಈ࡞֬ೝ༻ͷॲཧΛఆٛ͠΍͍͢ • ్தͰࢭΊͯ΋࣮ߦࡁΈͷfileλεΫ͸εΩοϓͯ͠ɺଓ ͖͔Β࣮ߦͰ͖Δ
9. ฒྻ࣮ߦ
10. ฒྻ࣮ߦ file1.txt ࡞੒ file2.txt ࡞੒ all.txt࡞੒ ͜͜͸ฒྻͰ࣮ߦͰ͖Δ
11. Pwrake https://github.com/masa16/Pwrake/ • ฒྻ࣮ߦ൛rake • ͦͷ··ͷRakefileͰฒྻ࣮ߦ͕Ͱ͖Δ • ࢦఆͨ͠λεΫʹґଘ͢ΔλεΫ܈ΛεϨουͰฒྻԽ ! • Πϯετʔϧํ๏
 $ gem install pwrake • ࢖͍ํ
 $ pwrake -j <ฒྻ਺> <λεΫ໊>
12. $ pwrake -j 2 all.txt << create file1.txt << create file2.txt echo 2222 > file2.txt >> complete file1.txt >> complete file2.txt << create all.txt cat file1.txt file2.txt > all.txt >> complete all.txt file "file1.txt" do |t| puts "<< create file1.txt" File.open(t.name, "w") do |file| file.puts("1111") end puts ">> complete file1.txt" end ! file "file2.txt" do |t| puts "<< create file2.txt" sh "echo 2222 > #{t.name}" puts ">> complete file2.txt" end ! file "all.txt" => ["file1.txt", "file2.txt"] do |t| ! puts "<< create all.txt" sh "cat #{t.prerequisites.join(' ')} 
 > #{t.name}" puts ">> complete all.txt" end
13. ϩάղੳͷ࣮૷ํ๏
14. ϩάղੳ access.01.log access.02.log access.03.log access.04.log result.01.csv result.02.csv result.03.csv result.04.csv result_all.csv
15. $ rake -P rake data/result.1.txt rake data/result.2.txt rake data/result.3.txt rake data/result_all.log data/result.1.txt data/result.2.txt data/result.3.txt ! $ pwrake -j 2 run # # log/access.1.log => data/result.1.log # log/access.2.log => data/result.2.log # log/access.3.log => data/result.3.log # results = [] (1..3).each do |num| logfile = "data/result.#{num}.txt" results << logfile ! file logfile do |t| sh "grep 'iPad' log/access.#{num}.log 
 > #{t.name}" end end ! # # data/result.1.log # data/result.2.log => data/result_all.log # data/result.3.log # file “data/result_all.log" => results do |t| sh "cat #{t.prerequisites.join(' ‘)} > 
 #{t.name}" end ! task :run => [“data/result_all.log"]
16. ͓·͚
17. Tips • fileλεΫͷλʔήοτ ͱ͢ΔϑΝΠϧ͸ɺॲཧ ͷ࠷ޙʹmvͰ࡞Δ • ్தͰఀࢭͨ͠ͱ͖ʹɺ ࡞Γ͔͚ϑΝΠϧ͕͋Δ ͨΊʹɺεΩοϓ͞Εͳ ͍Α͏ʹ͢ΔͨΊ file “target.txt” do |t| ! sh “grep HOGE src.log > “ +
 “ #{t.name}.tmp “ ! sh “mv #{t.name}.tmp #{t.name}” end