εΫϨΠϐϯάͷ҆ఆӡ༻ͷͨΊʹ ۤ࿑ͨ͠ͱ͜Ζɺ޻෉ͨ͠ͱ͜Ζ #BZTJEF
5FDI
#SJEHF


ΫϩʔϦϯάͷεϖγϟϦετ͕ޠΔɺΫϩʔϥʔӡ༻ͷཪଆʂ
ג ϏʔɾΞδϟΠϧ݉ɺδʔΫϥ΢υ ג
ࢤా
༟थ 

גࣜձࣾϏʔɾΞδϟΠϧ(΄΅ϑϦʔϥϯεɺ2012೥ΑΓ) δʔΫϥ΢υגࣜձࣾ औక໾ ݉೚ ϑϦʔϥϯε͕ू·ͬͯɺνʔϜ։ൃ डୗ։ൃ(8ׂ)ɺࣗࣾαʔϏε։ൃ(2ׂ) ϦʔϯɾελʔτΞοϓɺΞδϟΠϧ(اըɺ։ൃɺӡ༻) RailsɺSwiftɺ Java for Android ։ൃҊ݅΍ɺҰॹʹಇ͖͍ͨϑϦʔϥϯεͷํ େืूத!!! ຊൃදͰ͸ɺࣗࣾαʔϏεͷ҆ఆӡ༻ͷͨΊʹۤ ࿑ͨ͠ͱ͜Ζɺ޻෉ͨ͠ͱ͜ΖΛ͝঺հ 

ۤ࿑ͨ͠ͱ͜Ζ 

+BWB4DSJQU͕࣮ߦ͞Εͳ͍ͱ
৘ใ͕ͱΕͳ͍
αΠτଆͰ+4ͰಈతʹϖʔδΛߏஙͯ͠ΔͨΊ 

1PSUFSHFJTUͰεΫϨΠϐϯά Ruby Capybara Poltergeist PhantomJS ର৅αΠτ Safariͱಉ͡JSΤϯδϯ͕ಈ͘ Headlessϒϥ΢β CapybaraͷPhantomJSυϥΠό ड͚ೖΕςετ༻ςεςΟϯά ϑϨʔϜϫʔΫ 

require 'capybara/poltergeist' Capybara.register_driver :poltergeist do |app| Capybara::Poltergeist::Driver.new(app) end Capybara.default_driver = :poltergeist agent = Capybara.current_session agent.visit('URL') number = agent.find('CSSηϨΫλ').text.to_i 1PSUFSHFJTUͰεΫϨΠϐϯά 

Ϣʔβʔೝূ͠ͳ͍ͱ
৘ใ͕ͱΕͳ͍ 

agent.visit login_url agent.find('input[name="email"]').set(email) agent.find('input[name="pasword"]').set(password) agent.find('#login-btn').trigger('click') agent.visit account_url ೝূ͔ͯ͠Βର৅63-ʹΞΫηε 

ຖճೝূ͠௚͕͠ॏ͍ 

def save_cookie(agent, user) cookies_str = Base64.encode64( Marshal.dump( agent.driver.browser.cookies)) user.update_attributes(cookies: cookies_str) end def load_cookie(agent, user) cookies = Marshal.load( Base64.decode64(user.cookies)) cookies.values.each do |cookie| cookie_hash = JSON.parse(cookie.to_json) ["attributes"] agent.driver.browser.set_cookie(cookie_hash) end end $PPLJFʹΑΔೝূ 

DPPLJFͷ༗ޮظݶ͕
੾Ε͍ͯΔ͕࣌͋Δ 

DPPLJF͕੾Ε͍ͯͨΒSFUSZ scrape(need_login: true) do agent.visit('URL') agent.find('CSSηϨΫλ').text.to_i end # εΫϨΠϐϯά͢ΔՕॴ༻ڞ௨ϝιου def scrape need_login: false begin yield rescue => e if need_login && ! login? login retry end end end 

Կ౓΋ೝূτϥΠ͍ͯ͠Δͱ
ϩοΫ͞ΕΔ 

agent.driver.headers = { "User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36 #{Time.now.to_f.to_s}" } 6TFS"HFOUΛͪΐ͍ͪΐ͍ม͑Δ 

ΞΫηε͠·͍ͬͯ͘Δͱ
ϒϩοΫ͞ΕΔ 

Proxy1 (AWS) ΞϓϦ αʔό ର৅ αΠτ ΞΫηεݩΛͪΐ͍ͪΐ͍ม͑Δ Proxy1 (AWS) ᶃϒϩοΫ ᶄ৽͘͠ىಈ 

)5.-ߏ଄͕มΘΓ
εΫϨΠϐϯάࣦഊ
ͱ͔ 

"#ςετͯ͠ΔΒ͘͠
ΞΧ΢ϯτຖͰ)5.-͕ҧ͏ 

Ͱ͖Δ͚ͩදࣔܥ͔Β͸εΫϨΠϐϯά͠ͳ͍ ॓ധਓ਺ 9໊ ৘ใදࣔը໘ ॓ധਓ਺ 9 ϑΥʔϜը໘ ໊ ϑΥʔϜ෦෼ͷϚʔΫΞοϓ͸αʔόʔαΠυͷϓϩάϥϜͱ࿈ ܞ͍ͯ͠ΔͷͰมߋ͕ൃੜ͠ʹ͍͘

αΠτ͕ॏͯ͘
ͨ·ʹλΠϜΞ΢τͨ͠Γ
མͪͨΓ͢Δ 

ϩʔυ଴ͪɺදࣔ࣌ؒΛԆ͹͢ Capybara.register_driver :poltergeist do |app| Capybara::Poltergeist::Driver.new(app, :timeout => 60) end Capybara.default_driver = :poltergeist Capybara.default_max_wait_time = 30 agent = Capybara.current_session # ࠷େ60ඵ଴ͬͯ͘ΕΔ agent.visit('URL') # ࠷େ30ඵJavaScriptͷඇಉظߋ৽ͳͲͷऴྃΛ଴ͬͯ͘ΕΔ number = agent.find('CSSηϨΫλ').text.to_i 

εΫϨΠϐϯάࣦഊ͸
ඞ͓͖ͣΔ )5.-ߏ଄ͷมԽ ӬଓతΤϥʔ
઀ଓΤϥʔ Ұ࣌తͳΤϥʔ 

ϢχοτςετΛఆظతʹࣗಈ࣮ߦ project='ϦϙδτϦ໊' branch='master' api_token='APIτʔΫϯ' url=https://circleci.com/api/v1/project/${project}/ tree/${branch}?circle-token=${api_token} curl \ --header "Accept: application/json" \ --header "Content-Type: application/json" \ --request POST ${url} CircleCIͷϏϧυΛAPIΛ࢖ͬͯcron͔Βఆظ࣮ߦ ͚ͨ͜ΒCircleCI͕Slackʹ௨஌ͯ͘͠ΕΔ 

4JEFLJRͷར༻ # 1000ళฮ͋ͬͨͱͯ͠ shops.each do |shop| # ΋͠10ళฮ໨ͰΤϥʔͰॲཧ͕ͱ·ͬͯ͠·ͬͨΒ # 990ళฮ͕ະॲཧʹͳͬͯ͠·͏ shop.scrape end Ұ࣌తͳ઀ଓΤϥʔ ಛఆͷshopʹ͚ͩൃੜ͢Δ૝ఆ֎ͷΤϥʔ 

4JEFLJRͷར༻ ಛఆͷshopͰམͪͯ΋ɺଞͷshopͷॲཧ͸ݺ͹ΕΔ 

4JEFLJRͷͦͷଞར఺ ಛఆͷshopͰམͪͯ΋ɺଞͷshopͷॲཧ͸ݺ͹ΕΔ εϨουىಈ਺΍ىಈ·Ͱͷ଴ͪ࣌ؒΛ੍ޚ͠ɺର৅αΠτʹ ෛՙΛ͔͚ա͗ͳ͍ མͪͨ࣌ʹslackʹ௨஌ͯ͘͠ΕΔ མͪͨεϨουΛϦτϥΠͯ͘͠ΕΔ ࠷େϦτϥΠճ਺ࢦఆՄೳ ద౰ʹϦτϥΠִؒΛ޿͛ͳ͕Β(15, 16, 31, 96, 271, ... ) ϦτϥΠ࣌ͷslack௨஌ΛؒҾ͍ͨΓͰ͖Δ ฒྻॲཧ΍αʔόʔͷεέʔϧΞ΢τͰύϑΥʔϚϯεΞοϓ 

·ͱΊ ౰વͰ͖Δ͚ͩεΫϨΠϐϯάΤϥʔ΍ɺ઀ଓΤϥʔΛճආ͢ ΔΑ͏ʹ౒ྗ ͦΕͰ΋׬શʹ͸ճආͰ͖ͳ͍ εΫϨΠϐϯάΤϥʔΛ଎΍͔ʹݕ஌͢Δ࢓૊ΈΛ༻ҙ Ұ࣌తʹ઀ଓΤϥʔ΍ෆଌͷΤϥʔ͕ൃੜͨ͠৔߹͸ɺ֘౰ॲ ཧ͸ҟৗऴྃͭͭ͠΋ɺ༧ఆ͍ͯͨ͠όονॲཧ͸ܧଓ Ұ࣌తΤϥʔ(઀ଓΤϥʔɺcookie༗ޮظݶ੾Ε)͸ϦτϥΠ εΫϨΠϐϯάͷӡ༻ʹ͔͔Δ࣌ؒ΍ίετΛ͋Β͔͡ΊϓϩδΣ Ϋτॳظʹؔ܎ऀʹཧղͯ͠΋Β͏ඞཁ͕͋Δ(৔߹ʹΑͬͯ͸ Ϣʔβʔʹ΋) 

࠷ޙʹએ఻

ϦϯελΧϑΣ ΦϯϥΠϯ ϦʔϯɾελʔτΞοϓ(΍ͦͷଞྨࣅ)ख๏Λϕʔεʹͨ͠αʔ Ϗεاըɾ։ൃͷ࣮ફऀͷͨΊͷίϛϡχςΟ աڈ8ճͷΦϑϥΠϯษڧձΛ࣮ࢪ ΠϯλϏϡʔͷ࢓ํɺϢʔβʔςετͷ࢓ํɺMVPͷܾΊ ํɺࣾ಺ελʔτΞοϓͷۤ࿑࿩ɺͳͲͳͲ ͦΕͷΦϯϥΠϯ൛ ຖिਫ༵೔ 21:30 GoogleϋϯάΞ΢τʹͯ ݱࡏϝϯόʔ 4໊ ࢀՃऀ֤͕ࣗؔΘ͍ͬͯΔαʔϏεͷݱঢ়ใࠂ΍ɺ௚໘͍ͯ͠Δ ՝୊ͷڞ༗ͱ૬ޓΞυόΠεͱ͔ɺϦϯελܥͷຊͷಡॻձ ͝ڵຯ͋Ε͹੠Λ͔͚͍ͯͩ͘͞!