Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
スクレイピングは茨の道/Scraping is a thorny road
Search
cottondesu
December 17, 2018
Programming
0
170
スクレイピングは茨の道/Scraping is a thorny road
前回LT「Python + Selenium + Beautiful Soup でスクレイピング」の
1.おさらい
2.失敗から学ぶ
3.ソースの修正
4.結論
cottondesu
December 17, 2018
Tweet
Share
More Decks by cottondesu
See All by cottondesu
ヨシケイの取り忘れ対策した / Measures were taken to prevent forgetting to pick up Yoshikei
cottondesu
0
93
開発環境公開ハード編 / Development environment public hardware version
cottondesu
0
130
開発環境公開ソフト編 / Development environment public software edition
cottondesu
0
120
正規表現で心が折れた/Regular expressions broke my heart
cottondesu
0
140
オレオレGASからMakeへの移行を検討してみた / Considering the transition from Ore Ore GAS to Make
cottondesu
0
530
Kanazawa.rb 10周年KPT / Kanazawa.rb 10th Anniversary KPT
cottondesu
0
660
Kanazawa.rb 9周年KPT / Kanazawa.rb 9th Anniversary KPT
cottondesu
0
410
Mac Book Proのバッテリー交換してみた / I replaced the battery in my Mac Book Pro.
cottondesu
0
530
みんなのCSIRTコミュニティ紹介 / Everyone's CSIRT community introduction
cottondesu
0
420
Other Decks in Programming
See All in Programming
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
seike460
PRO
3
780
PipeCDのプラグイン化で目指すところ
warashi
1
280
生成AI時代のコンポーネントライブラリの作り方
touyou
1
230
XP, Testing and ninja testing
m_seki
3
250
ISUCON研修おかわり会 講義スライド
arfes0e2b3c
1
450
『自分のデータだけ見せたい!』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
akitotsukahara
2
640
チームのテスト力を総合的に鍛えて品質、スピード、レジリエンスを共立させる/Testing approach that improves quality, speed, and resilience
goyoki
5
920
テストから始めるAgentic Coding 〜Claude Codeと共に行うTDD〜 / Agentic Coding starts with testing
rkaga
13
4.7k
初学者でも今すぐできる、Claude Codeの生産性を10倍上げるTips
s4yuba
16
11k
Discover Metal 4
rei315
2
140
GitHub Copilot and GitHub Codespaces Hands-on
ymd65536
2
150
A full stack side project webapp all in Kotlin (KotlinConf 2025)
dankim
0
120
Featured
See All Featured
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Producing Creativity
orderedlist
PRO
346
40k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Raft: Consensus for Rubyists
vanstee
140
7k
We Have a Design System, Now What?
morganepeng
53
7.7k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
20k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
A Modern Web Designer's Workflow
chriscoyier
695
190k
Side Projects
sachag
455
42k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Transcript
2018年12月15日 kanazawa.rb meetup#76 εΫϨΠϐϯά ҵͷಓ
!DPUUPO@EFTV ΏΔͬͱ;ΘͬͱอकΤϯδχΞ ׂ৽ن։ൃɾۀվળ ׂอक ػೳՃɺػೳमਖ਼ɺόάमਖ਼ FUD
ΞδΣϯμ લճ-5ͷ͓͞Β͍ લճ-5ࣦഊ͔ΒֶͿ ιʔεͷमਖ਼ ݁
લճ-5ͷ͓͞Β͍
1ZUIPO 4FMFOJVN #FBVUJGVM4PVQͰεΫϨΠϐϯά
εΫϨΠϐϯάͱʁ
εΫϨΠϐϯά w ΣϒαΠτ͔ΒใΛநग़͢Δ ίϯϐϡʔλιϑτΣΞٕज़ w ݴޠ1FSMɺ1ZUIPOɺ3VCZɺ+BWB4DSJQUͰ ϥΠϒϥϦ͋Γ
શମͷߏਤ
શମͷߏਤ ϑΥϧμ EJSFOW WFOW QZϑΝΠϧ 4FMFOJVN XFCESJWFS ϩάΠϯϖʔδ $ISPNF
%FNP͕՚ྷʹ Τϥʔ
લճ-5ࣦഊ͔ΒֶͿ
ίʔυ֬ೝ
class main(): options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location = '/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser')
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main()
ҟৗऴྃ࣌ʹϓϩηε ͕Γଓ͚Δ ϔουϨεϒϥβͷͨΊΘ͔Γʹ͍͘
ղܾࡦ
ҟৗऴྃ࣌Ͱϓϩηε Λऴྃͤ͞Δඞཁ͕͋Δ
ιʔεͷमਖ਼
class main(): try: options = Options() # ChromeͷύεʢStableνϟωϧͰ--headless͕͑ΔΑ͏ʹͳͬͨΒෆཁͳͣʣ options.binary_location =
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' # ϔουϨεϞʔυΛ༗ޮʹ͢Δʢ࣍ͷߦΛίϝϯτΞτ͢Δͱը໘͕දࣔ͞ΕΔʣɻ options.add_argument('--headless') # ChromeͷWebDriverΦϒδΣΫτΛ࡞͢Δɻ driver = webdriver.Chrome( os.environ["CHROMEDRIVER"], chrome_options=options) # ϩάΠϯαΠτURL driver.get(os.environ["URL"]) # ϩάΠϯID driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd/div/input").send_keys( os.environ["LOGIN_USER_ID"]) # ϩάΠϯύεϫʔυ username = driver.find_element_by_xpath( "//div[@class='sub-content']/dl/dd[2]/div/input").send_keys( os.environ["LOGIN_USER_PASS"]) # ϩάΠϯϘλϯԡԼ driver.find_element_by_name("login").click() # ޱ࠲ཧʹભҠ driver.find_element_by_xpath(“//div[@id='link']/ul/li[10]").click() html = driver.page_source soup = BeautifulSoup(html, 'html.parser') मਖ਼Օॴ
# อ༗ޱɺऔಘ୯Ձɺऔಘ୯ՁɺධՁଛӹͷऔಘ item_name = ['อ༗ޱɹɹ |', 'औಘ୯Ձɹɹ |', 'ج४Ձֹɹɹ |',
'ධՁଛӹɹɹ |'] # ࢿ৴ୗ໊ͷऔಘ investmentname = [] investments = soup.find_all('td', class_='mbody', colspan="3") for investment in investments: investmentname.append(investment.a.text) inves_num = 0 row_num = 0 various_values = soup.find_all('tr', bgcolor='#eaf4e8', align="right") for various_value in various_values: # ࢿ৴ୗ໊ͷදࣔ print(investmentname[inves_num]) # อ༗ޱͷදࣔ print(item_name[row_num], various_value.td.text) for other in various_value.td.find_next_siblings("td"): row_num = row_num + 1 # อ༗ޱҎ֎ͷදࣔ print(item_name[row_num], other.text) row_num = 0 inves_num = inves_num + 1 except NoSuchElementException as e: print("seleniumͷૢ࡞தʹΤϥʔ͕ൃੜ͠·ͨ͠ɻ") traceback.print_exc() finally: # શͯͷΫοΩʔΛআ driver.delete_all_cookies() driver.quit() if __name__ == "__main__": main() मਖ਼Օॴ
݁
݁ w εΫϨΠϐϯάඞͣޭ͢ΔͱݶΒͳ͍ w ྫ֎ॲཧ USZFYDFQUpOBMMZ ͕ඞཁ w ͍ͭͰਖ਼ৗऴྃͤ͞Δॲཧ͕ඞཁ