我的PYTHON爬蟲編年史

Invited talk in Python Hsinchu User Group

Transcript

1. ౯ጱPYTHON粖恝翥ଙݥ Bingroom
2. About Me • Data engineer in • Comprehending DevOps • Addiction to and
4. C# PHP JAVA Python窕 أ㭏ၹ
5. Python窕 أ㭏ၹ
8. History Events
9. History Events Mechanize Selenium + Tkinter Web API PyQuery
10. Outline • Mechanize: web automation • Selenium (with Tkinter): build a tool to load dynamic
 content • Web API: find it, use it • PyQuery: fast crawler prototyping
11. Challenge 1: PTT
12. The Soup Everyone Loves
13. The Soup Everyone Loves
14. The Soup Everyone Loves
15. import mechanize
16. import mechanize
24. import mechanize Only in Python2
25. Python3: mechanicalsoup
26. Missing part in mechanicalsoup https://findbiz.nat.gov.tw/fts/query/QueryBar/queryInit.do
27. Missing part in mechanicalsoup
28. Missing part in mechanicalsoup
29. Challenge 2: 
 Facebook Personal IDs
30. Personal ID?
31. Personal ID: OK
32. How to query personal pages?
33. How to query personal pages? By Facebook ’s Vulnerability
34. How to deal with AJAX?
35. How to deal with AJAX? Selenium: for loading dynamic content
36. PhantomJS: headless browser
37. PhantomJS: headless browser Headless Chrome
38. PIL: crop reCAPTCHA
39. PhantomJS: headless browser Headless Chrome PIL: crop reCAPTCHA Tkinter: GUI design
40. PhantomJS: headless browser Headless Chrome cx_Freeze: everything to executable PIL: crop reCAPTCHA Tkinter: GUI design
41. Good Game
42. Challenge 3: 
 22 Government Official Sites
43. News & Attachment 岄玖૱(tpe)物https://health.gov.taipei/Default.aspx?tabid=36&mid=442 岄Ӿ૱(txg)物 http://www.health.taichung.gov.tw/26216/26204/26216/26204/26201/26204/27056/Lpsimplelist चᵇ૱(klu)物 http://www.klchb.gov.tw/ch/news/newspaper/list.aspx?c0=640 岄ܖ૱(tnn)物 https://health.tainan.gov.tw/list.asp?nsub=A0A600&topage=1 ṛᵜ૱(khh)物 http://khd.kcg.gov.tw/Main.aspx?sn=398 碝玖૱(ntpe)物 http://www.health.ntpc.gov.tw ਩葦翻(ilc)物 http://www.ilshb.gov.tw/index.php?catid=23&cid=4 ໘瑼૱(tyc)物 http://dph.tycg.gov.tw/home.jsp?id=13 ࡱ嬝૱(cyi)物 https://www.cichb.gov.tw/news/indexpda.asp 碝ᒓ翻(hsh)物 http://www.hcshb.gov.tw/home.jsp?mserno=200802220002&serno=200802220015&menudata=HcshbMenu& 碝ᒓ૱(hsc)物 http://www.hccg.gov.tw/MunicipalNews?websitedn=ou=hcchb,ou=ap_root,o=hccg,c=tw&language=chinese 舏礣翻(mal)物 https://www.mlshb.gov.tw/tc/PressRelease.aspx?pn=1&department=7 ܖಭ翻(nto)物 http://www.ntshb.gov.tw/Default.aspx ୼玕翻(cwh)物 http://www.chshb.gov.tw/news/?type_id=127&top=0 襇຋翻(yun)物 http://www.ylshb.gov.tw/news/index.php?m=9&m1=14&m2=35 ࡱ嬝翻(chy)物 https://cyshb.cyhg.gov.tw/News.aspx?n=E236E03D6AE796F8&sms=A55ECAF6D99EACF8 ੽䩚翻(pch)物 http://www.ptshb.gov.tw/News.aspx?CategorySN=1894&n=CC774672B906BC5E 臺荳翻(hwa)物 http://www.hlshb.gov.tw/files/40-1006-27-1.php 岄䩚翻(ttt)物 http://www.ttshb.gov.tw/files/40-1000-12-1.php?Lang=zh-tw ᰂ槹翻(kmn)物 http://phb.kinmen.gov.tw/News.aspx?n=87FB8DD4C759A8B5&sms=A2C62D68901B977C ᄯ竝翻(peh)物 https://www.phchb.gov.tw/home.jsp?id=23 蝫࿯翻(lnn)物 http://www.matsuhb.gov.tw/2009web/news/news_contents.php?room=news1
44. News & Attachment 岄玖૱(tpe)物https://health.gov.taipei/Default.aspx?tabid=36&mid=442 岄Ӿ૱(txg)物 http://www.health.taichung.gov.tw/26216/26204/26216/26204/26201/26204/27056/Lpsimplelist चᵇ૱(klu)物 http://www.klchb.gov.tw/ch/news/newspaper/list.aspx?c0=640 岄ܖ૱(tnn)物 https://health.tainan.gov.tw/list.asp?nsub=A0A600&topage=1 ṛᵜ૱(khh)物 http://khd.kcg.gov.tw/Main.aspx?sn=398 碝玖૱(ntpe)物 http://www.health.ntpc.gov.tw ਩葦翻(ilc)物 http://www.ilshb.gov.tw/index.php?catid=23&cid=4 ໘瑼૱(tyc)物 http://dph.tycg.gov.tw/home.jsp?id=13 ࡱ嬝૱(cyi)物 https://www.cichb.gov.tw/news/indexpda.asp 碝ᒓ翻(hsh)物 http://www.hcshb.gov.tw/home.jsp?mserno=200802220002&serno=200802220015&menudata=HcshbMenu& 碝ᒓ૱(hsc)物 http://www.hccg.gov.tw/MunicipalNews?websitedn=ou=hcchb,ou=ap_root,o=hccg,c=tw&language=chinese 舏礣翻(mal)物 https://www.mlshb.gov.tw/tc/PressRelease.aspx?pn=1&department=7 ܖಭ翻(nto)物 http://www.ntshb.gov.tw/Default.aspx ୼玕翻(cwh)物 http://www.chshb.gov.tw/news/?type_id=127&top=0 襇຋翻(yun)物 http://www.ylshb.gov.tw/news/index.php?m=9&m1=14&m2=35 ࡱ嬝翻(chy)物 https://cyshb.cyhg.gov.tw/News.aspx?n=E236E03D6AE796F8&sms=A55ECAF6D99EACF8 ੽䩚翻(pch)物 http://www.ptshb.gov.tw/News.aspx?CategorySN=1894&n=CC774672B906BC5E 臺荳翻(hwa)物 http://www.hlshb.gov.tw/files/40-1006-27-1.php 岄䩚翻(ttt)物 http://www.ttshb.gov.tw/files/40-1000-12-1.php?Lang=zh-tw ᰂ槹翻(kmn)物 http://phb.kinmen.gov.tw/News.aspx?n=87FB8DD4C759A8B5&sms=A2C62D68901B977C ᄯ竝翻(peh)物 https://www.phchb.gov.tw/home.jsp?id=23 蝫࿯翻(lnn)物 http://www.matsuhb.gov.tw/2009web/news/news_contents.php?room=news1
45. https://aji.tw/slides/pycon2017
46. https://aji.tw/slides/pycon2017
47. PyQuery == jQuery • parent element > child element • # for id • . for class • . when a class name has space (i.e. has 2 classes)
49. #table_0 > tbody > tr:nth-child(1) > td.CCMS_jGridView_td_Class_1 Chrome Inspector
50. Firefox Inspector (Firebug) #table_0 > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2)
52. Usage (1) Generate DOM object by URL opener
53. Usage (2) Traverse elements and list target items
54. Usage (2) Traverse elements and list target items
55. Usage (3) Laugh.
56. Usage (3) Laugh. #table_0 > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2) ( ) td ጱ ᒫԫ㮆 ጱ text
57. Usage (3) Laugh. #table_0 > tbody:nth-child(2) > tr:nth-child(1) > td:nth-child(2) ( ) td ጱ ᒫԫ㮆 a ጱ 痀௔ href
58. Usage (3) Laugh.
59. Beautifulsoup V.S. PyQuery
60. Preparation for timeout issue
61. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn)
62. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Ӟ෭玖ṛ
63. 䌃ਠᵍॠፗ矑硬粚 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn)
64. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) encoding=Big5
65. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) Date አ࿆㾴ଙ
66. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) 猂硬አ࿆㾴ଙ
67. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) ֦磪ݣ傀㰷独㻟牫 猂硬አ࿆㾴ଙ
68. 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) / 22 https://github.com/bingroom/city-health-news-TW
69. https://github.com/bingroom/city-health-news-TW 8 / 22 FAILED: 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn)
70. https://github.com/bingroom/city-health-news-TW 岄玖૱(tpe) 岄Ӿ૱(txg) चᵇ૱(klu) 岄ܖ૱(tnn) ṛᵜ૱(khh) 碝玖૱(ntpe) ਩葦翻(ilc) ໘瑼૱(tyc) ࡱ嬝૱(cyi) 碝ᒓ翻(hsh) 碝ᒓ૱(hsc) 舏礣翻(mal) ܖಭ翻(nto) ୼玕翻(cwh) 襇຋翻(yun) ࡱ嬝翻(chy) ੽䩚翻(pch) 臺荳翻(hwa) 岄䩚翻(ttt) ᰂ槹翻(kmn) ᄯ竝翻(peh) 蝫࿯翻(lnn) 22 / 22 Successful:
71. Challenge 4: 
 Hack Nielsen DAR System
72. How to get API from a website?
73. • Guess How to get API from a website?
74. How to get API from a website? • Guess
75. • Guess • Inspect How to get API from a website?
76. • Guess • Inspect How to get API from a website?
77. • Guess • Inspect How to get API from a website?
78. Nielsen DAR Ad
79. Nielsen DAR Ad • Login by mechanize
80. Nielsen DAR Ad • Hold the header
81. Nielsen DAR Ad • Enjoy(?) the API
82. • Facebook • Gmail Inbox • Dcard • PChome Graph API Multiprocess OAuth2.0 Xml base64 JSON regex random UA Keyword for Web APIs
83. Summary • Mechanize • BeautifulSoup • Selenium + Phantomjs • APIs
 Less documentation
 Less limitation • Mechanicalsoup • PyQuery • Selenium + Chromedriver • APIs
 More documentation
 More limitation Past Now
85. PTT GOV websites Pchome Dcard Facebook Almost everything!
86. PTT GOV websites Pchome Dcard Facebook Almost everything!
87. Sikuli: PTT Crawler
88. Sikuli: Game Farming
89. Thanks for your listening! httpstatusdogs.com