Scraping with Geb

GR8Conf Eu 2016

Transcript

1. June 1st - 3rd 2016 Copenhagen, Denmark SCRAPING WITH GEB
2. SERGIO DEL AMO ME@SERGIODELAMO.COM @SDELAMO
4. IOS APP APP STORE
5. GR8CONFAGENDA GOOGLEPLAY
6. GROOVYCALAMARI.COM A “weekly” curated email newsletter full of interesting, relevant links about the Groovy Ecosystem
7. GEB HTTP://GEBISH.ORG
8. http://www.webbotsspidersscreenscrapers.com
9. WHAT CAN YOU DO? ▸ PRICE-MONITORING WEBBOTS ▸ IMAGE-CAPTURING WEBBOTS ▸ LINK VERIFICATION WEBBOTS ▸ WEBBOTS THAT SEND EMAIL ▸ WEBBOTS THAT CONVERT A WEBSITE IN AN API ▸ SNIPERS
10. EXAMPLE 1 CREW SCRAPING HTTPS://GITHUB.COM/SDELAMO/GEBWEBBOT_GR8CONF
11. GR8CONF API ▸ Only for agenda, no sponsors or crew
12. java -jar gebwebbot_gr8conf-all.jar crew|sponsors destinationFolder outputFilename sqlite|plist|csv phantomjs.binary.path
13. GRADLE SHADOW HTTPS://GITHUB.COM/JOHNRENGELMAN/SHADOW
15. DESIRED OUTPUT
16. MODEL
17. MODEL
18. TEST YOUR FETCHER
19. SPOCK - TEST YOUR FETCHER
22. Define the interesting parts of your pages in a concise, maintanable and extensible manner GEB PAGES
23. DIV.CREW DIV.CREW DIV.CREW DIV.CREW DIV.CREW DIV.CREW DIV.CREW DIV.CREW DIV.CREW
25. Modules are re-usable definitions of content that can be used across multiple pages GEB MODULES
27. OUTPUT
30. EXAMPLE 2 SPONSORS SCRAPING HTTPS://GITHUB.COM/SDELAMO/GEBWEBBOT_GR8CONF
31. GR8CONF API ▸ Only for agenda, no sponsors or crew
32. DIV.SPONSORS DIV.SPONSORS H4 DIV.SPONSOR DIV.SPONSOR DIV.SPONSOR H4
33. DIV.SPONSORS DIV.SPONSORS H4 DIV.SPONSOR DIV.SPONSOR DIV.SPONSOR H4
34. DIV.SPONSORS DIV.SPONSORS H4 DIV.SPONSOR DIV.SPONSOR DIV.SPONSOR H4
35. DIV.SPONSORS DIV.SPONSORS H4 DIV.SPONSOR DIV.SPONSOR DIV.SPONSOR H4
38. GEB EXAMPLE GRADLE HTTPS://GITHUB.COM/GEB/GEB-EXAMPLE-GRADLE The following commands will launch the tests with the individual browsers: ./gradlew chromeTest ./gradlew firefoxTest ./gradlew phantomJsTest To run with all, you can run: ./gradlew test MARCIN ERDMANN
39. GEB.CONFIG
40. EXAMPLE 3 PAGINATION HTTPS://GITHUB.COM/SDELAMO/WEBBOT_GEB_MEETUP_MEMBERS
42. DYNAMIC URL http://www.meetup.com/es-ES/madrid-gug/members/49149882/ BASE URL: MEETUP GROUP SLUG: MEMBER ID: http://www.meetup.com/ madrid-gug 28938802
44. PAGINATION .PAGINATION .NAV-NEXT
45. PAGINATION
47. PAGINATION MODULE
48. HARVEST AND VISIT
49. HARVEST LINKS
51. SPLIT LOAD BETWEEN WEBBOTS HTTPS://HTTPSTATUSDOGS.COM
52. SPLIT LOAD BETWEEN WEBBOTS 1 2 5 3 4 11 12 15 13 14 21 22 25 23 24 31 32 35 33 34 41 42 45 43 44 6 7 10 8 9 16 17 20 18 19 26 27 30 28 29 36 37 40 38 39 46 47 50 48 49 def ids = 1..50 def webbotIndex = 3 def webbotsInParallel = 6 int total = ids.size() def sublistsSize = (total / webbotsInParallel) as int def s = ids.collate(sublistsSize)[webbotIndex]
53. USER AGENT SPOOFING
54. USER AGENT SPOOFING HTTPS://GITHUB.COM/SDELAMO/GEBWEBBOT_USERAGENT
55. USER AGENT SPOOFING
56. EXAMPLE 4 HIDDEN CONTENT AND ON MOUSE OVER EVENTS
59. FAILS: HIDDEN CONTENT
60. CALL A JS METHOD
61. MOVE TO ELEMENT
62. INCLUDE LIBRARY
63. UI INTERACTION
64. KEYBOARD
65. SLIDERS
66. STEALTH MEANS SIMULATING HUMAN PATTERNS ▸ BE KIND TO YOUR RESOURCES ▸ RUN YOUR WEBBOTS DURING BUSY HOURS ▸ DON’T RUN YOUR WEBBOTS AT THE SAME TIME EACH DAY ▸ DON’T RUN YOUR WEBBOT ON HOLIDAYS AND WEEKENDS ▸ USE RANDOM, INTRA-FETCH DELAYS
67. SIMULATE HUMAN CLICK RHYTHM
68. ?