Puppeteer Fetchbot

Library and Shell command that provides a simple JSON-API to perform human like interactions and data extractions on any website. Built on top of puppeteer.

Transcript

1. a Node library which provides an API to control Chrome and how we used it to automate UI-interactions 1 . 1
2. Bernhard Behrendt // developer at 1 . 2
3. programming in... 1 . 3
4. and doing my work with... 1 . 4
5. in a distributed scrum team of eight people Dresden Wiesbaden Marburg Köln 1 . 5
6. 1 . 6
7. which are two shop systems 1. an internal one // accessibility driven 2. a public one // marketing driven 1 . 7
8. 2 . 1
9. npm install puppeteer 2 . 2
10. const puppeteer = require('puppeteer'); 2 . 3
11. const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); })(); 2 . 4
12. const puppeteer = require('puppeteer'); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); await page.goto('https://aoe.com'); await page.screenshot({path: 'aoe.png'}); await browser.close(); })(); 2 . 5
13. 2 . 6
14. 2 . 7
15. const puppeteer = require('puppeteer'); const SELECTOR_GOOGLE_SEARCH_INPUT = 'input'; const SELECTOR_GOOGLE_SEARCH_RESULT = '#rso > div > div > div:nth const SELECTOR_AOE_OSS_LINK = '#page-header > div.header-menus > (async () => { const browser = await puppeteer.launch({headless:false}); const page = await browser.newPage(); page.setViewport({ width:1024, height:768 }); 2 . 8
16. const puppeteer = require('puppeteer'); const SELECTOR_GOOGLE_SEARCH_INPUT = 'input'; (async () => { const browser = await puppeteer.launch({headless: false}); const page = await browser.newPage(); page.setViewport({ width: 1024, height: 768 }); await page goto('https://google com'); 2 . 9
17. { results: [ 'AOE: Agile Softwareentwicklung | IT & Technologie Dienstlei '//SEIBERT/MEDIA, Wiesbaden - Agile Software-Entwicklung, At 'Scholz & Volkmer', '– 3deluxe - Transdisciplinary Design –', 'vitronic.de - VITRONIC - the machine vision people', 'AbbVie: Startseite | Deutschland', 'AOE – the open web company | LinkedIn', 'Chemical Company and Manufacturer - The WeylChem Group', 'Sweco.de | Sweco.de', 'Braun + Company' ] } 2 . 10
18. Element Amount Text inputs 43 Selects (2 clicks each) 2-4 Checkboxes 12 Radio Button Sets 10 Datepicker 1 Buttons 11 Total min. 56 – max. ca. 90 3 . 1
19. 3 . 2
20. But... who's playing with that puppet... 4 . 1
21. ...Fetchbot does... 4 . 2
22. npm install fetchbot 4 . 3
23. URL::ACTION::SELECTOR 4 . 4
24. 4 . 5
25. { "http://google.com":{ "root":true, } } 4 . 6
26. { "http://google.com":{ "root":true, "type":[["input", "web companies in wiesbaden\n"]], "waitFor":[[3000]], } } 4 . 7
27. { "http://google.com":{ "root":true, "type":[["input", "web companies in wiesbaden\n"]], "waitFor":[[3000]], "fetch":{ "h3.r as headlines":[], "#rso > div > div > div > div > div > h3 > a as links":{ "attr": "href", "type": [] } } } 4 . 8
28. fetchbot --job fetch.json --agent "LONG_LONG_USER_AGENT_STRING" const FetchBot = require('fetchbot'); (async () => { const fetchbot = new FetchBot(''); let fetchBotData = await fetchbot.runAndExit('./fetch.json'); console.log(fetchBotData); })(); 4 . 9
29. { "headlines": [ "AOE: Agile Softwareentwicklung | IT & Technologie Dienst "//SEIBERT/MEDIA, Wiesbaden - Agile Software-Entwicklung, "Scholz & Volkmer", "[...]" ], "links": [ "https://www.aoe.com/de/home.html", "https://www.seibert-media.net/", "https://www.s-v.de/", "[...]" ] } 4 . 10
30. 4 . 11
31. const fetchbot = new FetchBot('', { "attached": true, "slowmo": 250, "width": 1280, "height": 1024, "trust": true }); 4 . 12
32. Type casting Boolean Number String Array of String(s) Array of Numbers(s) Objects containing an additional attribute matching 4 . 13
33. 4 . 14
34. 4 . 15
35. { "https://testserver.local/PostForms/webshopform.php": { "root": true, "click": "input[type='submit']", "waitFor": [[5000]] }, "https://testserver.local/tarifuebersicht-kabel": { "click": "#sg-133 > div > div.cobra-product-footer > p > butt "waitFor": [[1000]] }, "https://testserver.local/breitband/produktkonfiguration/": [ { 4 . 16
36. ✏ Variant Duration manual 2 minutes automated 1 minute 4 . 17
37. Integration as a lib 4 . 18
38. 4 . 19
39. 4 . 20
40. 4 . 21
41. "Who's able to write JSON can use fetchbot" 4 . 22
42. 4 . 23
43. Thank you for listening 5
44. Icon made by SimpleIcon from www. aticon.com 6