Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scraping the Web with AWS Lambda and PhantomJS

Scraping the Web with AWS Lambda and PhantomJS

A talk given at Greater Philadelphia AWS User Group meetup on May 25, 2016. The talk summarizes our experience creating a scalable website scraper and the many iterations of technology we went through to achieve our final product.

You can find the source code of PhantomJS/Node.js web scraper for AWS Lambda at https://github.com/akrylysov/lambda-phantom-scraper.

Artem Krylysov

May 25, 2016

Other Decks in Programming


  1. Scraping the Web with AWS Lambda and PhantomJS Artem Krylysov

  2. A look at the experience of creating a scalable website

    scraper and the many iterations of technology we went through to achieve our final product.
  3. blockbust.io

  4. blockbust.io Blockbust is a service which scans a website and

    helps to identify if any of its HTML elements may be blocked by an ad blocker like Adblock, Adblock Plus, uBlock or many others. Blockbust fetches the content of the web page and checks it against the list of known rules.
  5. blockbust.io

  6. First attempt

  7. First attempt The Blockbust backend is written in Go, the

    first decision was to use the easiest and the fastest way - http.Get. http.Get makes an HTTP request to a web server and you can read the response.
  8. First attempt It was very fast but didn't work well.

    Almost all modern websites use JavaScript to generate some additional content.
  9. First attempt For example, for facebook.com, the size of the

    initial HTML page returned by the server is 300 KB. If you open the same page in a browser and wait until it’s completely loaded, the size of the HTML content doubles to 600 KB.
  10. PhantomJS

  11. PhantomJS PhantomJS is a headless version of Chromium browser. It

    provides a JavaScript API for navigating web pages, interacting with a DOM and taking screenshots. http://phantomjs.org/
  12. PhantomJS var page = require('webpage').create(); page.open('http://phantomjs.org/documentation/', function(status) { if (status

    === 'success') { var title = page.evaluate(function() { return document.title; }); console.log('Title: ' + title); phantom.exit(0); } else { phantom.exit(1); } });
  13. PhantomJS $ phantomjs example1.js Title: Documentation | PhantomJS

  14. PhantomJS PhantomJS is not a Node.js module, it is a

    standalone application. If you want to use it from your Node.js script you have to execute the phantomjs binary as a child process. You can communicate with it using stdin and stdout.
  15. PhantomJS We use phantomjs-prebuilt package, it provides an easy way

    to install PhantomJS binaries using NPM and use it from Node.js. https://github.com/Medium/phantomjs
  16. Docker and ECS

  17. Docker and ECS ECS is a service for dunning and

    managing Docker containers on EC2 cluster.
  18. Docker and ECS Didn’t provide a way to scale the

    number of instances of a specified Docker container automatically. You could place your EC2 container cluster into an auto scaling group and it would add a new EC2 instance to the cluster, but it wouldn’t launch a new Docker container.
  19. Queue

  20. Queue Our next option was to add a queue in

    front of the scraper and limit the number of concurrent requests. In this case, if we had a huge spike in traffic, users would have to wait in a line for their results and it might take a long time. We wanted to avoid such kind of bad user experience.
  21. AWS Lambda

  22. AWS Lambda We already had an experience of using AWS

    Lambda in a few internal projects: • Amazon S3 event handlers. • Amazon SNS notification handlers. • Slack bots.
  23. What is Lambda? AWS Lambda is a service which runs,

    manages and automatically scales your code.
  24. AWS Lambda Supports Node.js, Python and Java.

  25. AWS Lambda You can run Lambda: • As a handler

    for S3, SNS, CloudWatch and many other AWS events. • On a schedule using cron-like syntax. • As an HTTP request handler using API Gateway.
  26. AWS Lambda exports.handler = function(event, context, callback) { callback(null, 'hello');

    } handler - the main function which is called by AWS Lambda. event - AWS uses this parameter to pass the event data (e.g. POST data for API Gateway). callback - you can use this parameter to return the data.
  27. Testing Lambda

  28. Testing Lambda Amazon doesn't provide any tools for testing lambdas

    on a local development environment. You can easily emulate the Lambda runtime using a simple Express. js server.
  29. Testing Lambda var express = require('express'); var bodyParser = require('body-parser');

    var lambda = require('./lambda'); var app = express(); app.use(bodyParser.json()); app.post('/', function(req, res) { lambda.handler(req.body, {}, function(err, result) { if (err) { return res.send(err); } res.send(result); }); }); app.listen(3000);
  30. Deploying

  31. Deploying First of all, to deploy your code on Amazon

    servers you need to build a deployment package. The deployment package is a ZIP archive which contains the application code and the node_modules directory if you have any dependencies.
  32. Deploying You can deploy the package using the web interface

    or using awscli tool: aws lambda update-function-code \ --region us-east-1 \ --function-name lambda-phantom-scraper \ --zip-file fileb://$PWD/lambda-phantom-scraper.zip
  33. Tweaking PhantomJS

  34. Tweaking PhantomJS We successfully moved our code to Lambda, and

    stopped seeing any stability or performance issues, but sometimes we still were getting inaccurate results for some websites.
  35. Tweaking PhantomJS: issue 1 The default stdout buffer size in

    Node.js is 200 KB. Use maxBuffer to increase the buffer size when you create a child process: var phantom = childProcess.execFile(phantomJsPath, childArgs, { env: {URL: url}, maxBuffer: 2048*1024 });
  36. Tweaking PhantomJS: issue 2 In PhantomJS the default screen size

    is 400 to 300 pixels, that makes some websites think that our scraper is a mobile browser. You can change the screen size using viewportSize property: page.viewportSize = {width: 1366, height: 768};
  37. Tweaking PhantomJS: issue 3 The default PhantomJS user agent is

    Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1. Change it to a user agent from a real browser: page.settings.userAgent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36';
  38. Lambda limitations

  39. Debugging Lambda Amazon doesn’t provide any ways to debug your

    code. You have to use console.log like in early JavaScript days.
  40. Lambda Limits The maximum number of concurrent executions is 100

    by default. You need to ask the AWS support if you want to increase the limit.
  41. Lambda cold start The execution time is usually from 1

    to 4 seconds longer for a cold start - a moment when AWS internally spins a new “container” for your application. It happens e.g. when the function wasn’t called in a while or when AWS needs to scale your Lambda function. There is no solution for this problem at the moment.
  42. How much does it cost?

  43. How much does it cost? The Lambda cost depends on

    3 major factors: • Number of executions • Amount of allocated memory • Execution time
  44. How much does it cost? 128MB - $0.000000208 per 100ms

    512MB - $0.000000834 per 100ms 1024MB - $0.000001667 per 100ms 1536MB - $0.000002501 per 100ms Full pricing table https://aws.amazon.com/lambda/pricing/.
  45. How much does it cost? We configured our Lambda function

    to use 1GB of memory. In average, a website takes about 4 seconds to load. For 100000 calls we payed: 0.000001667 * 40 * 100000 = $6.668
  46. Conclusions

  47. Lambda Cons • Cold start latency • Debugging

  48. Lambda Pros • Don’t pay for it if you don’t

    use it • Easy to manage and deploy • Scales automatically
  49. Questions?

  50. Thanks! https://github.com/akrylysov/lambda-phantom-scraper