Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクレイピングは茨の道/Scraping is a thorny road
Search
cottondesu
December 17, 2018
Programming
0
130
スクレイピングは茨の道/Scraping is a thorny road
前回LT「Python + Selenium + Beautiful Soup でスクレイピング」の
1.おさらい
2.失敗から学ぶ
3.ソースの修正
4.結論
cottondesu
December 17, 2018
Tweet
Share
More Decks by cottondesu
See All by cottondesu
正規表現で心が折れた/Regular expressions broke my heart
cottondesu
0
25
オレオレGASからMakeへの移行を検討してみた / Considering the transition from Ore Ore GAS to Make
cottondesu
0
340
Kanazawa.rb 10周年KPT / Kanazawa.rb 10th Anniversary KPT
cottondesu
0
450
Kanazawa.rb 9周年KPT / Kanazawa.rb 9th Anniversary KPT
cottondesu
0
290
Mac Book Proのバッテリー交換してみた / I replaced the battery in my Mac Book Pro.
cottondesu
0
400
みんなのCSIRTコミュニティ紹介 / Everyone's CSIRT community introduction
cottondesu
0
310
他人事ではないサイドチャネル攻撃 / Side-channel attacks are no stranger to us.
cottondesu
0
160
Kanazawa.rbコミュニティ紹介 / Kanazawa.rb community introduction
cottondesu
0
120
swagger頑張ったー/Good work on the swagger.
cottondesu
0
230
Other Decks in Programming
See All in Programming
Ruby Function Composition
bkuhlmann
1
330
Java 22 Overview
kishida
1
170
Netty Chicago Java User Group 2024-04-17
sullis
0
140
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
26
7.9k
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
130
#phpcon_odawara オープン・クローズドなテストフィクスチャを求めて / open closed test fixtures
77web
3
220
スクラムチームと認知負荷 - ニフティのスクラムトーク Vol2. / NIFTY Tech Talk #18
niftycorp
PRO
1
120
Elm 0.19.0 Changes
bkuhlmann
0
490
DMMプラットフォームがTiDB Cloudを採用した背景
pospome
8
3.7k
Folding Cheat Sheet #2
philipschwarz
PRO
0
110
Folding Cheat Sheet #1
philipschwarz
PRO
0
210
Designing for tomorrow's programming workflows
honnibal
PRO
2
110
Featured
See All Featured
GraphQLとの向き合い方2022年版
quramy
31
12k
Statistics for Hackers
jakevdp
789
220k
Automating Front-end Workflow
addyosmani
1355
200k
Rebuilding a faster, lazier Slack
samanthasiow
72
8.2k
The Brand Is Dead. Long Live the Brand.
mthomps
48
28k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
13
1.5k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
29
6k
A better future with KSS
kneath
231
16k
The Cost Of JavaScript in 2023
addyosmani
14
3.8k
Visualization
eitanlees
135
14k
GitHub's CSS Performance
jonrohan
1023
450k
Side Projects
sachag
451
41k
Transcript
2018年12月15日 kanazawa.rb meetup#76 εΫϨΠϐϯά ҵͷಓ
!DPUUPO@EFTV ΏΔͬͱ;ΘͬͱอकΤϯδχΞ ׂ৽ن։ൃɾۀվળ ׂอक ػೳՃɺػೳमਖ਼ɺόάमਖ਼ FUD
ΞδΣϯμ લճ-5ͷ͓͞Β͍ લճ-5ࣦഊ͔ΒֶͿ ιʔεͷमਖ਼ ݁
લճ-5ͷ͓͞Β͍
1ZUIPO 4FMFOJVN #FBVUJGVM4PVQͰεΫϨΠϐϯά
εΫϨΠϐϯάͱʁ
εΫϨΠϐϯά w ΣϒαΠτ͔ΒใΛநग़͢Δ ίϯϐϡʔλιϑτΣΞٕज़ w ݴޠ1FSMɺ1ZUIPOɺ3VCZɺ+BWB4DSJQUͰ ϥΠϒϥϦ͋Γ
શମͷߏਤ
શମͷߏਤ ϑΥϧμ EJSFOW WFOW QZϑΝΠϧ 4FMFOJVN XFCESJWFS ϩάΠϯϖʔδ $ISPNF
%FNP͕՚ྷʹ Τϥʔ
લճ-5ࣦഊ͔ΒֶͿ
ίʔυ֬ೝ
class main(): options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location = '/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser')
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main()
ҟৗऴྃ࣌ʹϓϩηε ͕Γଓ͚Δ ϔουϨεϒϥβͷͨΊΘ͔Γʹ͍͘
ղܾࡦ
ҟৗऴྃ࣌Ͱϓϩηε Λऴྃͤ͞Δඞཁ͕͋Δ
ιʔεͷमਖ਼
class main(): try: options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location =
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser') मਖ਼Օॴ
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 except NoSuchElementException as e: print("seleniumͷૢ࡞தʹΤϥʔ͕ൃੜ͠·ͨ͠ɻ") traceback.print_exc() finally: # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main() मਖ਼Օॴ
݁
݁ w εΫϨΠϐϯάඞͣޭ͢ΔͱݶΒͳ͍ w ྫ֎ॲཧ USZFYDFQUpOBMMZ ͕ඞཁ w ͍ͭͰਖ਼ৗऴྃͤ͞Δॲཧ͕ඞཁ