PhantomJS for Web Page Automation @ariyahidayat Mountain View Jan 8, 2014 South Bay Selenium Meetup

/usr/bin/whoami shapesecurity.com

“Software Provocateur” PhantomJS Esprima

11-Second Elevator Pitch Scriptable Simulate user interactions Headless Command-line, no GUI WebKit Layout and rendering

Important Traits Multi platforms Zero dependencies (*) Works with Other Tools

Who is Using PhantomJS http://phantomjs.org/users.html

Versions vs Downloads http://phantomjs.org/releases.html Announced Jan 23, 2011 WebDriver Winter 2012 Pure headless Linux Spring 2012

Web Page Processing Screen Capture Headless Testing Network Monitoring

5-Minute Primer

Hello, World! console.log('Hello, world!'); phantom.exit();

Capture to Image var page = require('webpage').create(); page.open('http://google.com', function (status) { page.render('google.png'); phantom.exit(); });

Get Page Title var page = require('webpage').create(); page.open('http://linkedin.com', function (status) { if (status !== 'success') { console.log('FAIL to load the address'); } else { var title = page.evaluate(function () { return document.title; }); console.log('Page title is', title); } phantom.exit(); });

Use jQuery (or Other Libs) var page = require('webpage').create(); page.open('http://www.sample.com', function() { page.includeJs("http://code.jquery.com/jquery-1.10.2.js", function() { page.evaluate(function() { $("button").click(); }); phantom.exit() }); });

Behind the Scene C++ Framework Operating Systems WebKit PhantomJS Ghost Driver

Web Page Processing

Ghost Driver https://speakerdeck.com/detronizator/boston Ghost Driver is a pure JavaScript implementation of the WebDriver Wire Protocol for PhantomJS. It's a Remote WebDriver that uses PhantomJS as back-end. https://github.com/detro/ghostdriver

Python Example from selenium import webdriver driver = webdriver.PhantomJS('phantomjs') driver.get('http://www.phantomjs.org') print driver.title driver.quit() pip install selenium

PhantomJS (Remote) WebDriver phantomjs --webdriver=9134 Port

Ruby Example require "selenium-webdriver" driver = Selenium::WebDriver.for(:remote, :url => "http://localhost:9134") driver.navigate.to "http://google.com" element = driver.find_element(:name, 'q') element.send_keys "PhantomJS" element.submit puts driver.title driver.quit gem install selenium-webdriver

Node.js JavaScript Example var wd = require('wd'); var driver = wd.remote('localhost', 9134); driver.init(function() { driver.get("http://casperjs.org", function() { driver.title(function(err, title) { console.log(title); }); }); }); npm install wd

.NET and C# See http://michael-whelan.net/using-phantomjs-with-webdriver http://www.andykelk.net/tech/headless-browser-testing-with-phantomjs- selenium-webdriver-c-nunit-and-mono

No content

Using CasperJS var casper = require('casper').create(); casper.start('http://duckduckgo.com', function() { this.echo(this.getTitle()); }); casper.then(function() { this.fill('form', { q: 'Selenium San Jose' }, true); }); casper.then(function() { var snippet = this.evaluate(function() { return document.querySelector('div#links > div').textContent; }); this.echo(snippet); }); casper.run();

Headless Testing

No content

On Being Defensive http://ariya.ofilabs.com/2012/12/quality-code-via-multiple-layers-of-defense.html

Pre-commit Hook http://ariya.ofilabs.com/2012/03/git-pre-commit-hook-and-smoke-testing.html if [ `date +%w` -eq 6 ]; then echo "Enjoy your life. Do not work on Sunday!" exit 1 fi exit 0 make test

Jasmine Mocha QUnit YUITest Capybara tapedeck Karma Testem Buster.JS Travis CI Grunt Laika

“YOU SHALL NOT PASS!” — Darth Vader

Screen Capture

HTML-to-Anything var page = require('webpage').create(); page.open(title, function (status) { if (status !== 'success') { console.log('FAIL to load the address'); } else { page.render(filename); } phantom.exit(); }); Outputs: external file, base64 string Formats: PNG, JPEG, GIF, PDF

Rendering Aspects Zoom Factor Scale the page up and down Viewport Size Simulates different window dimensions Clipping Rectangle Crop portions of the page

Media Queries, Responsive Design http://mediaqueri.es/

Dichromacy http://ariya.ofilabs.com/2012/10/web-page-screenshot-with-phantomjs.html

De-CSS-ify (No Stylesheets) http://ariya.ofilabs.com/2013/06/capturing-web-page-without-stylesheets.html

Resources Blocking page.onResourceRequested = function(requestData, request) { if ((/http:\/\/.+?\.css$/gi).test(requestData['url'])) { console.log('Skipping', requestData['url']); request.abort(); } }; Skipping http://static.bbci.co.uk/frameworks/barlesque/2.45.9/mobile/3.5/style/main.css Skipping http://static.bbci.co.uk/bbcdotcom/0.3.184/style/mobile/bbccom.css Skipping http://static.bbci.co.uk/news/1.7.1-259/stylesheets/core.css Skipping http://static.bbci.co.uk/news/1.7.1-259/stylesheets/compact.css

Visual Regression http://tldr.huddle.com/blog/css-testing/ https://github.com/Huddle/PhantomCSS https://github.com/facebook/huxley https://github.com/bslatkin/dpxdt

Network Monitoring

Requests + Responses Capture var page = require('webpage').create(); page.onResourceRequested = function(request) { console.log('Request ' + JSON.stringify(request, undefined, 4)); }; page.onResourceReceived = function(response) { console.log('Receive ' + JSON.stringify(response, undefined, 4)); }; page.open(url);

Network Waterfall

YSlow + PhantomJS http://yslow.org/phantomjs/

Future Outlook

Engine vs Browser Pratt & Whitney JT9D http://ariya.ofilabs.com/2011/06/your-webkit-port-is-special-just-like-every-other-port.html

Diversity = Better

Future (with Caveat Emptor) PhantomJS 2.x Fresher, multi-process QtWebKit PhantomJS 3.x Chromium-based

Thank You @ariyahidayat ariya.ofilabs.com/highlights speakerdeck.com/ariya Some artworks are from http://openclipart.org.