Web Scraping with Node: Exposing crawler data by API.

Transcript

1. Web Scraping with Node Exposing crawler data by API.
2. Who am I Rafael Almeida (Lycan) - @RafaellLycan Front-End Engineer at Tripda. Red Bull, RPG, Star Wars, Lego.
3. Schedule What is Web Scraping/Crawler? Why we do that? How we do that in NodeJS? Our Target Code (Meh.)
4. What is Web Scraping
5. What is Web Scraping Web scraping is a computer software technique of extracting information from websites.
6. What is Web Scraping Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines.
7. WEB SITES STALKER.
9. Why we do that
10. Why we do that To Challenge ourselves To get more accessible data To improve our hacking skills To have fun
11. WHAT IS YOUR MOTIVATION?
12. https://github.com/rafaell-lycan/sabesp-mananciais-api
13. How do that in Node
14. WHY NODE?
16. Package JSON { "name": "star-wars-scraping", "version": "1.0.0", "author": "Rafaell Lycan <sonny.webdsg@gmail.com>", }
17. Our great Dependencies Express Request Cheerio Bluebird
18. Install Dependencies npm install —save express request cheerio bluebird
19. Our Target
20. http://www.starwars.com/films
21. Using Express var express = require('express'); var app = express(); app.get('/', function (req, res) { res.json({message:'Star Wars Scraping API’}); }); app.listen(3000);
22. Using Express //=========== var routes = require('./routes'); app.use('/', routes); app.use(function(err, req, res, next) { res.status(400).json({ error: err }); }); //===========
23. Using Express var express = require('express'); var router = express.Router(); router.get('/', function(req, res, next) { res.json({message:'Star Wars Scraping API'}); }); module.exports = router;
24. Using Request var request = require('request'); request(url, function (error, response, body){ if(error){ return throw new Error(); } // Get body and do something nice! :) }
25. Using Cheerio var cheerio = require('cheerio'); var $ = cheerio.load(body);
26. Using Bluebird var Promise = require('bluebird');
 module.exports = { getTitle :function(url) { return new Promise(function(resolve, reject) { request(url, function (error, response, html) { if (error) { reject(error); } var $ = cheerio.load(html); var title = $(‘title').text(); resolve(title); } }
 }
27. Using Bluebird //=========== var request = Promise.promisify(require('request')); //=========== getTitle :function(url) { return request(url).then(function(result){ var html = result[1];
 var $ = cheerio.load(html);
 var title = $(‘title’).text(); return title; } } //===========
28. WE REALY NEED TO DO CRAWLER ALWAYS?
29. STORE THE DAMN DATA!
30. Install Mongo npm install —save mongojs
31. Using Mongo var mongo = require('mongojs'); var db = mongo(‘star-wars-movies'); module.exports = db.collection('movies');
32. Using Mongo var Collection = Promise.promisifyAll(require('./Movies’)); Collection.findAsync().then(...); Collection.insertAsync().then(...);
33. MAKE YOUR DATA ACCESSIBLE!
34. WHAT THE FUC#!
35. WHAT THE FUC#!
36. Install CORS npm install —save cors
37. Using CORS //=========== var cors = require('cors'); app.use(cors()); //===========
39. IN GOD WE TRUST. ALL OTHERS MUST BRING DATA!
42. Install Google Analytics npm install —save universal-analytics
43. Using Google Analytics var ua = require('universal-analytics'); var Analytics = { visitor = ua('UA-XXXX-XX'), track : function() { this.visitor.pageview('/').send(); } }; module.exports = Analytics;
44. Using Google Analytics //=========== var ga = require(‘./analytics‘); app.use(ga.track); //===========
45. BITCH PLEASE SHOW ME THE DAMN CODE!
46. Thats All Folks ABOUT ME: /rafaell.lycan GITHUB: http://bit.ly/star-wars-crawler