Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

厲害了，我的蟲 - 我在iii的爬蟲人生

Avatar for bingroom bingroom
June 02, 2018
Programming
530
0

厲害了，我的蟲 - 我在iii的爬蟲人生

Lightning talk in PyConTW 2018

Avatar for bingroom

bingroom

June 02, 2018

More Decks by bingroom

See All by bingroom
我的PYTHON爬蟲編年史
Avatar for bingroom bingroom
0
220
我如何在PyCon找到乾爹 ─ 工人智慧的展現
Avatar for bingroom bingroom
0
260

Other Decks in Programming

See All in Programming
全ての永続トークンを、生まれる前に消し去りたい！
Avatar for sylph01 sylph01
2
980
[ハンズオン]AIへの指示だけで「五目並べ」を作ってみよう
Avatar for Satoshi Kaneyasu satoshi256kbyte
1
380
[Dev Group Paris Octobre 26] Agentforce, SIforce, Claudeforce, Slackbot… on s'y retrouve ?
Avatar for Nathan Abondance nabondance
0
110
Findy - エンジニア向け会社紹介/Findy Company Deck
Avatar for Findy findyinc
6
400k
Agentic Software Factoryに、すごく賢いIF文を。 / super smart IF statement into the Agentic Software Factory.
Avatar for r-kagaya rkaga
8
2.3k
JIS keyboard support in Omarchy
Avatar for LEF lef237
0
130
気持ちぃ〜角丸 Squircle
Avatar for ryu ryu1013
0
320
ChromeでローカルLLM
Avatar for ryu ryu1013
0
140
APNsからLive Activityを開始する話
Avatar for Ryotaro Onoue yumnumm
0
170
設計次第でAIコードの読む量は減らせる / designing-for-code-reading
Avatar for MinoDriven minodriven
35
17k
Optunaを作ったときにやりたかったこと、その後 (2026/10/16, Optuna Meetup #3)
Avatar for Takuya Akiba iwiwi
1
440
phperだってパンチカードでコーディングしたい
Avatar for hideki kinjyo o0h
PRO
0
350

Featured

See All Featured
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
520
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
409
67k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
53k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.6k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
New Earth Scene 8
Avatar for Sydney Stone popppiees
4
2.7k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
3
1.5k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
41k
Un-Boring Meetings
Avatar for Sebastian Deterding codingconduct
0
460
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
192
12k
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
2
310

Transcript

  1. ʒ ౯ࣁ iii ጱ粖恝Ոኞ Bingroom

  2. ʒ ౯ࣁ iii ጱ粖恝Ոኞ Bingroom

  3. Me

  4. Python أ㭏

  5. C PHP JAVA

  6. None
  7. None
  8. None

  9. ض拻奾抷

  10. for i in iii :

  11. for i in iii : make_crawler(i)

  12. for i in iii : make_crawler(i) if i == "

    ":

  13. for i in iii : make_crawler(i) if i == "

    ": break

  14. 粖恝Ոኞ EP1 Be a Contributor

  15. Ӟ෭癨۽ in ◦◦ੴ

  16. beautifulsoup

  17. Best Editor ever

  18. None

  19. Find Global IDs

  20. [email protected]

  21. [email protected]

  22. [email protected]

  23. None
  24. None
  25. None

  26. selenium

  27. + phantomjs + tkinter + cx_freeze selenium

  28. Good Game

  29. य़疑ጱ虻碘ਞ獊ࠩ!

  30. 粖恝Ոኞ EP2 Be a Hacker

  31. PChome 24h / mall

  32. Dcard

  33. Mechanize <form> </form>

  34. Mechanize <form> </form>

  35. Crawl their own systems

  36. ؃ጭ獈 Mechanize

  37. 玲header ؃ጭ獈 Mechanize

  38. https://mediaview.nielsen.com/mediaview/ocr/messagebroker/json/hzService/lookup? qArg.entitledReports=OcrReportBuilder%2CXcrReportBuilder %2COcrCrossCampaignReportBuilder%2COcrFlexFormatterReportBuilder %2COcrCrossCampaignFlexFormatterReportBuilder %2C&queryNamespace=com.nielsen.media.mediaView.ocr.campaignDetailsMapper&query=get CampaignAudienceProfile&qArg.pv_cmpg_id=600748601&qArg.pv_imponly_mode=N&qArg. pv_all_cmpg=N&qArg.pv_dim_media_pln_ids=-1&qArg.pv_dev_typ=1%2C2%2C3&qArg.pv_ frequency_distribution=-1&qArg.pv_reporting_frequency=100&qArg.pv_creative_type=-1&qA rg.pv_start_dt_dt=7-Jul-2017&qArg.pv_end_dt_dt=14- Aug-2017&qArg.pv_viewable_mode=N&qArg.pv_clct_strt_dt=7-

    Jul-2017&qArg.pv_is_vwbl_demo_flg=N&qArg.pv_is_ott_flag=Y&qArg.pv_connected_device _flag=N&qArg.pv_geo_ids=1222%24&qArg.pv_request_id=1505387203299_twmedia.contract

  39. 粖恝Ոኞ EP3 Be an Engineer

  40. iii: 犋膏࿆凗ڥ

  41. ૡ纷䒍๜蛪 犋膏࿆凗ڥ

  42. ᎣՈᎣᶎ

  43. 犋ඪ萇

  44. 觬ߝਞ獊蝢䁭羬翄

  45. 觬ߝਞ獊蝢䁭羬翄

  46. 岄玖૱(tpe)物https://health.gov.taipei/Default.aspx?tabid=36&mid=442 岄Ӿ૱(txg)物 http://www.health.taichung.gov.tw/26216/26204/26216/26204/26201/26204/27056/Lpsimplelist चᵇ૱(klu)物 http://www.klchb.gov.tw/ch/news/newspaper/list.aspx?c0=640 岄ܖ૱(tnn)物 https://health.tainan.gov.tw/list.asp?nsub=A0A600&topage=1 ṛᵜ૱(khh)物 http://khd.kcg.gov.tw/Main.aspx?sn=398 碝玖૱(ntpe)物

    http://www.health.ntpc.gov.tw ਩葦翻(ilc)物 http://www.ilshb.gov.tw/index.php?catid=23&cid=4 ໘瑼૱(tyc)物 http://dph.tycg.gov.tw/home.jsp?id=13 ࡱ嬝૱(cyi)物 https://www.cichb.gov.tw/news/indexpda.asp 碝ᒓ翻(hsh)物 http://www.hcshb.gov.tw/home.jsp?mserno=200802220002&serno=200802220015&menudata=Hcs 碝ᒓ૱(hsc)物 http://www.hccg.gov.tw/MunicipalNews?websitedn=ou=hcchb,ou=ap_root,o=hccg,c=tw&languag 舏礣翻(mal)物 https://www.mlshb.gov.tw/tc/PressRelease.aspx?pn=1&department=7 ܖಭ翻(nto)物 http://www.ntshb.gov.tw/Default.aspx ୼玕翻(cwh)物 http://www.chshb.gov.tw/news/?type_id=127&top=0 襇຋翻(yun)物 http://www.ylshb.gov.tw/news/index.php?m=9&m1=14&m2=35 ࡱ嬝翻(chy)物 https://cyshb.cyhg.gov.tw/News.aspx?n=E236E03D6AE796F8&sms=A55ECAF6D99EACF8 ੽䩚翻(pch)物 http://www.ptshb.gov.tw/News.aspx?CategorySN=1894&n=CC774672B906BC5E 臺荳翻(hwa)物 http://www.hlshb.gov.tw/files/40-1006-27-1.php 岄䩚翻(ttt)物 http://www.ttshb.gov.tw/files/40-1000-12-1.php?Lang=zh-tw ᰂ槹翻(kmn)物 http://phb.kinmen.gov.tw/News.aspx?n=87FB8DD4C759A8B5&sms=A2C62D68901B977C ᄯ竝翻(peh)物 https://www.phchb.gov.tw/home.jsp?id=23 蝫࿯翻(lnn)物 http://www.matsuhb.gov.tw/2009web/news/news_contents.php?room=news1 碝股ᑤ牏獍ޞ牏ᴫկ

  47. 岄玖૱(tpe)物https://health.gov.taipei/Default.aspx?tabid=36&mid=442 岄Ӿ૱(txg)物 http://www.health.taichung.gov.tw/26216/26204/26216/26204/26201/26204/27056/Lpsimplelist चᵇ૱(klu)物 http://www.klchb.gov.tw/ch/news/newspaper/list.aspx?c0=640 岄ܖ૱(tnn)物 https://health.tainan.gov.tw/list.asp?nsub=A0A600&topage=1 ṛᵜ૱(khh)物 http://khd.kcg.gov.tw/Main.aspx?sn=398 碝玖૱(ntpe)物

    http://www.health.ntpc.gov.tw ਩葦翻(ilc)物 http://www.ilshb.gov.tw/index.php?catid=23&cid=4 ໘瑼૱(tyc)物 http://dph.tycg.gov.tw/home.jsp?id=13 ࡱ嬝૱(cyi)物 https://www.cichb.gov.tw/news/indexpda.asp 碝ᒓ翻(hsh)物 http://www.hcshb.gov.tw/home.jsp?mserno=200802220002&serno=200802220015&menudata=Hcs 碝ᒓ૱(hsc)物 http://www.hccg.gov.tw/MunicipalNews?websitedn=ou=hcchb,ou=ap_root,o=hccg,c=tw&languag 舏礣翻(mal)物 https://www.mlshb.gov.tw/tc/PressRelease.aspx?pn=1&department=7 ܖಭ翻(nto)物 http://www.ntshb.gov.tw/Default.aspx ୼玕翻(cwh)物 http://www.chshb.gov.tw/news/?type_id=127&top=0 襇຋翻(yun)物 http://www.ylshb.gov.tw/news/index.php?m=9&m1=14&m2=35 ࡱ嬝翻(chy)物 https://cyshb.cyhg.gov.tw/News.aspx?n=E236E03D6AE796F8&sms=A55ECAF6D99EACF8 ੽䩚翻(pch)物 http://www.ptshb.gov.tw/News.aspx?CategorySN=1894&n=CC774672B906BC5E 臺荳翻(hwa)物 http://www.hlshb.gov.tw/files/40-1006-27-1.php 岄䩚翻(ttt)物 http://www.ttshb.gov.tw/files/40-1000-12-1.php?Lang=zh-tw ᰂ槹翻(kmn)物 http://phb.kinmen.gov.tw/News.aspx?n=87FB8DD4C759A8B5&sms=A2C62D68901B977C ᄯ竝翻(peh)物 https://www.phchb.gov.tw/home.jsp?id=23 蝫࿯翻(lnn)物 http://www.matsuhb.gov.tw/2009web/news/news_contents.php?room=news1 碝股ᑤ牏獍ޞ牏ᴫկ

  48. 岄玖૱(tpe)物https://health.gov.taipei/Default.aspx?tabid=36&mid=442 岄Ӿ૱(txg)物 http://www.health.taichung.gov.tw/26216/26204/26216/26204/26201/26204/27056/Lpsimplelist चᵇ૱(klu)物 http://www.klchb.gov.tw/ch/news/newspaper/list.aspx?c0=640 岄ܖ૱(tnn)物 https://health.tainan.gov.tw/list.asp?nsub=A0A600&topage=1 ṛᵜ૱(khh)物 http://khd.kcg.gov.tw/Main.aspx?sn=398 碝玖૱(ntpe)物

    http://www.health.ntpc.gov.tw ਩葦翻(ilc)物 http://www.ilshb.gov.tw/index.php?catid=23&cid=4 ໘瑼૱(tyc)物 http://dph.tycg.gov.tw/home.jsp?id=13 ࡱ嬝૱(cyi)物 https://www.cichb.gov.tw/news/indexpda.asp 碝ᒓ翻(hsh)物 http://www.hcshb.gov.tw/home.jsp?mserno=200802220002&serno=200802220015&menudata=Hcs 碝ᒓ૱(hsc)物 http://www.hccg.gov.tw/MunicipalNews?websitedn=ou=hcchb,ou=ap_root,o=hccg,c=tw&languag 舏礣翻(mal)物 https://www.mlshb.gov.tw/tc/PressRelease.aspx?pn=1&department=7 ܖಭ翻(nto)物 http://www.ntshb.gov.tw/Default.aspx ୼玕翻(cwh)物 http://www.chshb.gov.tw/news/?type_id=127&top=0 襇຋翻(yun)物 http://www.ylshb.gov.tw/news/index.php?m=9&m1=14&m2=35 ࡱ嬝翻(chy)物 https://cyshb.cyhg.gov.tw/News.aspx?n=E236E03D6AE796F8&sms=A55ECAF6D99EACF8 ੽䩚翻(pch)物 http://www.ptshb.gov.tw/News.aspx?CategorySN=1894&n=CC774672B906BC5E 臺荳翻(hwa)物 http://www.hlshb.gov.tw/files/40-1006-27-1.php 岄䩚翻(ttt)物 http://www.ttshb.gov.tw/files/40-1000-12-1.php?Lang=zh-tw ᰂ槹翻(kmn)物 http://phb.kinmen.gov.tw/News.aspx?n=87FB8DD4C759A8B5&sms=A2C62D68901B977C ᄯ竝翻(peh)物 https://www.phchb.gov.tw/home.jsp?id=23 蝫࿯翻(lnn)物 http://www.matsuhb.gov.tw/2009web/news/news_contents.php?room=news1 碝股ᑤ牏獍ޞ牏ᴫկ
  49. None
  50. None
  51. None

  52. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn)

  53. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Ӟ෭玖ṛ

  54. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) 䌃ਠᵍॠፗ矑硬粚

  55. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) encoding=Big5

  56. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Date አ࿆㾴ଙ

  57. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Date አ࿆㾴ଙ

  58. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Date አ࿆㾴ଙ ֦磪ݣ傀㰷独㻟牫

  59. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) / 22 https://github.com/bingroom/city-health-news-TW

  60. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) 8 / 22 FAILED: https://github.com/bingroom/city-health-news-TW

  61. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh)

    碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) 22 / 22 Successful: https://github.com/bingroom/city-health-news-TW

  62. ptt gov websites pchome dcard facebook almost everything

  63. 粖恝Ոኞ 綛क़缰 Be a Farmer

  64. አ PCMAN 粖 ptt

  65. ᛔ㵕಑ௗ

  66. ᛔ㵕搱蝿瞁發

  67. None
  68. None

  69. 粖恝Ոኞ ਠ奾缰

  70. None

  71. ॠ఺

  72. None

  73. ፡憎ԧӞ螇ط

  74. None

  75. THE END