Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクレイピングは茨の道/Scraping is a thorny road
Search
cottondesu
December 17, 2018
Programming
0
170
スクレイピングは茨の道/Scraping is a thorny road
前回LT「Python + Selenium + Beautiful Soup でスクレイピング」の
1.おさらい
2.失敗から学ぶ
3.ソースの修正
4.結論
cottondesu
December 17, 2018
Tweet
Share
More Decks by cottondesu
See All by cottondesu
ヨシケイの撮り忘れを対策したPart2/measures-were-taken-to-prevent-forgetting-to-pick-up-yoshikei-part2
cottondesu
0
19
ヨシケイの取り忘れ対策した / Measures were taken to prevent forgetting to pick up Yoshikei
cottondesu
0
120
開発環境公開ハード編 / Development environment public hardware version
cottondesu
0
140
開発環境公開ソフト編 / Development environment public software edition
cottondesu
0
130
正規表現で心が折れた/Regular expressions broke my heart
cottondesu
0
150
オレオレGASからMakeへの移行を検討してみた / Considering the transition from Ore Ore GAS to Make
cottondesu
0
550
Kanazawa.rb 10周年KPT / Kanazawa.rb 10th Anniversary KPT
cottondesu
0
680
Kanazawa.rb 9周年KPT / Kanazawa.rb 9th Anniversary KPT
cottondesu
0
420
Mac Book Proのバッテリー交換してみた / I replaced the battery in my Mac Book Pro.
cottondesu
0
540
Other Decks in Programming
See All in Programming
アルテニア コンサル/ITエンジニア向け 採用ピッチ資料
altenir
0
110
Cache Me If You Can
ryunen344
2
3k
MCPでVibe Working。そして、結局はContext Eng(略)/ Working with Vibe on MCP And Context Eng
rkaga
5
2.3k
🔨 小さなビルドシステムを作る
momeemt
4
690
Improving my own Ruby thereafter
sisshiki1969
1
160
AI時代のUIはどこへ行く?
yusukebe
18
9k
OSS開発者という働き方
andpad
5
1.7k
RDoc meets YARD
okuramasafumi
4
170
AWS発のAIエディタKiroを使ってみた
iriikeita
1
190
testingを眺める
matumoto
1
140
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
120
時間軸から考えるTerraformを使う理由と留意点
fufuhu
16
4.8k
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
49
14k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Raft: Consensus for Rubyists
vanstee
140
7.1k
Embracing the Ebb and Flow
colly
87
4.8k
Side Projects
sachag
455
43k
Agile that works and the tools we love
rasmusluckow
330
21k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.2k
BBQ
matthewcrist
89
9.8k
Documentation Writing (for coders)
carmenintech
74
5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Optimizing for Happiness
mojombo
379
70k
Rails Girls Zürich Keynote
gr2m
95
14k
Transcript
2018年12月15日 kanazawa.rb meetup#76 εΫϨΠϐϯά ҵͷಓ
!DPUUPO@EFTV ΏΔͬͱ;ΘͬͱอकΤϯδχΞ ׂ৽ن։ൃɾۀվળ ׂอक ػೳՃɺػೳमਖ਼ɺόάमਖ਼ FUD
ΞδΣϯμ લճ-5ͷ͓͞Β͍ લճ-5ࣦഊ͔ΒֶͿ ιʔεͷमਖ਼ ݁
લճ-5ͷ͓͞Β͍
1ZUIPO 4FMFOJVN #FBVUJGVM4PVQͰεΫϨΠϐϯά
εΫϨΠϐϯάͱʁ
εΫϨΠϐϯά w ΣϒαΠτ͔ΒใΛநग़͢Δ ίϯϐϡʔλιϑτΣΞٕज़ w ݴޠ1FSMɺ1ZUIPOɺ3VCZɺ+BWB4DSJQUͰ ϥΠϒϥϦ͋Γ
શମͷߏਤ
શମͷߏਤ ϑΥϧμ EJSFOW WFOW QZϑΝΠϧ 4FMFOJVN XFCESJWFS ϩάΠϯϖʔδ $ISPNF
%FNP͕՚ྷʹ Τϥʔ
લճ-5ࣦഊ͔ΒֶͿ
ίʔυ֬ೝ
class main(): options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location = '/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser')
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main()
ҟৗऴྃ࣌ʹϓϩηε ͕Γଓ͚Δ ϔουϨεϒϥβͷͨΊΘ͔Γʹ͍͘
ղܾࡦ
ҟৗऴྃ࣌Ͱϓϩηε Λऴྃͤ͞Δඞཁ͕͋Δ
ιʔεͷमਖ਼
class main(): try: options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location =
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser') मਖ਼Օॴ
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 except NoSuchElementException as e: print("seleniumͷૢ࡞தʹΤϥʔ͕ൃੜ͠·ͨ͠ɻ") traceback.print_exc() finally: # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main() मਖ਼Օॴ
݁
݁ w εΫϨΠϐϯάඞͣޭ͢ΔͱݶΒͳ͍ w ྫ֎ॲཧ USZFYDFQUpOBMMZ ͕ඞཁ w ͍ͭͰਖ਼ৗऴྃͤ͞Δॲཧ͕ඞཁ