Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Web (Web Scraping - Parte 2)

A Web (Web Scraping - Parte 2)

More Decks by Code4Pol - Reprogramando a Política

Other Decks in Technology

Transcript

  1. WWW

  2. TIM BERNERS-LEE Firstly, it discusses the problem of information access

    at CERN. Then, it introduces the idea of linked information systems, and compares them with less flexible ways of finding information. https:/ /www.w3.org/History/1989/proposal.html “
  3. Human-readable information linked together in an unconstrained way. Multimedia documents

    which include graphics, speech and video. HYPERTEXT HYPERMEDIA WWW
  4. TEXTO IMAGEM VIDEO SOM APLICAÇÃO Sistema de documentos em hipermídia

    que são interligados e executados na Internet
  5. MARKUP LANGUAGES Mussum Ipsum Cacilds vidis litro abertis. Praesent malesuada

    urna nisi, quis volutpat erat hendrerit non. Nam vulputate dapibus. Praesent vel viverra nisi. Mauris aliquet nunc non turpis scelerisque, eget. • Suco de cevadiss deixa as pessoas mais interessantiss. • Todo mundo vê os porris que eu tomo • Ninguém vê os tombis que eu levo! Si u mundo tá muito paradis? Toma um mé que o mundo vai girarzis! Viva Forevis aptent taciti sociosqu ad litora torquent Não sou faixa preta cumpadi, sou preto inteiris, inteiris. Mais vale um bebadis conhecidiss, que um alcoolatra anonimiss.
  6. MARKUP LANGUAGES Mussum Ipsum Cacilds vidis litro abertis. Praesent malesuada

    urna nisi, quis volutpat erat hendrerit non. Nam vulputate dapibus. Praesent vel viverra nisi. Mauris aliquet nunc non turpis scelerisque, eget. • Suco de cevadiss deixa as pessoas mais interessantiss. • Todo mundo vê os porris que eu tomo • Ninguém vê os tombis que eu levo! Si u mundo tá muito paradis? Toma um mé que o mundo vai girarzis! Viva Forevis aptent taciti sociosqu ad litora torquent Não sou faixa preta cumpadi, sou preto inteiris, inteiris. Mais vale um bebadis conhecidiss, que um alcoolatra anonimiss. TÍTULO
  7. MARKUP LANGUAGES Mussum Ipsum Cacilds vidis litro abertis. Praesent malesuada

    urna nisi, quis volutpat erat hendrerit non. Nam vulputate dapibus. Praesent vel viverra nisi. Mauris aliquet nunc non turpis scelerisque, eget. • Suco de cevadiss deixa as pessoas mais interessantiss. • Todo mundo vê os porris que eu tomo • Ninguém vê os tombis que eu levo! Si u mundo tá muito paradis? Toma um mé que o mundo vai girarzis! Viva Forevis aptent taciti sociosqu ad litora torquent Não sou faixa preta cumpadi, sou preto inteiris, inteiris. Mais vale um bebadis conhecidiss, que um alcoolatra anonimiss. SUBLINHADO
  8. MARKUP LANGUAGES Mussum Ipsum Cacilds vidis litro abertis. Praesent malesuada

    urna nisi, quis volutpat erat hendrerit non. Nam vulputate dapibus. Praesent vel viverra nisi. Mauris aliquet nunc non turpis scelerisque, eget. • Suco de cevadiss deixa as pessoas mais interessantiss. • Todo mundo vê os porris que eu tomo • Ninguém vê os tombis que eu levo! Si u mundo tá muito paradis? Toma um mé que o mundo vai girarzis! Viva Forevis aptent taciti sociosqu ad litora torquent Não sou faixa preta cumpadi, sou preto inteiris, inteiris. Mais vale um bebadis conhecidiss, que um alcoolatra anonimiss. LISTA
  9. MARKUP LANGUAGES Mussum Ipsum Cacilds vidis litro abertis. Praesent malesuada

    urna nisi, quis volutpat erat hendrerit non. Nam vulputate dapibus. Praesent vel viverra nisi. Mauris aliquet nunc non turpis scelerisque, eget. • Suco de cevadiss deixa as pessoas mais interessantiss. • Todo mundo vê os porris que eu tomo • Ninguém vê os tombis que eu levo! Si u mundo tá muito paradis? Toma um mé que o mundo vai girarzis! Viva Forevis aptent taciti sociosqu ad litora torquent Não sou faixa preta cumpadi, sou preto inteiris, inteiris. Mais vale um bebadis conhecidiss, que um alcoolatra anonimiss. NEGRITO
  10. IMG TEXTO … Sistema de documentos em hipermídia que são

    interligados e executados na Internet AVI
  11. html header body p div span ul li li li

    title img HTML ÁRVORE DOM document object model
  12. lab-20161007-webscraping-2 $ python -m http.server 8888 Serving HTTP on 0.0.0.0

    port 8888 ... lab-20161007-webscraping-2 $ python -m SimpleHttpServer 8888 Serving HTTP on 0.0.0.0 port 8888 ... PYTHON 3 PYTHON 2
  13. $ python scrapestepbystep.py b'<html>\n\n\t<head>\n\t\t<title>Minha Pagina</title>\n\t</head>\n\n\t<body>\n\t \t<p>Oi Pessoal!!!</p>\n\n\t\t<div>\n\t\t\t \t<span>Eu sou o:</span>\n\t\t\t\t<ul>\n\t\t

    \t\t\t<li>Alexandre</li>\n\t\t\t\t \t<li>Aleluia</li>\n\t\t\t\t\t<li>Alecrim</ li>\n\t\t\t\t</ul>\n\n\t\t\t<img src="https://cdn4.iconfinder.com/data/icons/ reaction/32/happy-128.png"/>\n\t</div>\n\t</ body>\n\n</html>'
  14. $ python Python 3.5.0 (default, Dec 10 2015, 21:32:09) [GCC

    4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import html >>> import requests >>> page = requests.get("http://localhost:8888/index.html") >>> tree = html.fromstring(page.content) >>> print(tree) <Element html at 0x10eaf7d68>
  15. $ python Python 3.5.0 (default, Dec 10 2015, 21:32:09) [GCC

    4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import html >>> import requests >>> page = requests.get("http://localhost:8888/index.html") >>> tree = html.fromstring(page.content) >>> print(tree) <Element html at 0x10eaf7d68> >>> print(tree.tag) html
  16. $ python Python 3.5.0 (default, Dec 10 2015, 21:32:09) [GCC

    4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import html >>> import requests >>> page = requests.get("http://localhost:8888/index.html") >>> tree = html.fromstring(page.content) >>> print(tree) <Element html at 0x10eaf7d68> >>> print(tree.tag) html >>> len(tree) 2 >>
  17. $ python Python 3.5.0 (default, Dec 10 2015, 21:32:09) [GCC

    4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import html >>> import requests >>> page = requests.get("http://localhost:8888/index.html") >>> tree = html.fromstring(page.content) >>> print(tree) <Element html at 0x10eaf7d68> >>> print(tree.tag) html >>> len(tree) 2 >>> tree[0] <Element head at 0x10eaf7ef8> >>> tree[1] <Element body at 0x10eaf7db8> >>> for node in tree: ... print(node) ... <Element head at 0x10eaf7ef8> <Element body at 0x10eaf7db8>
  18. $ python Python 3.5.0 (default, Dec 10 2015, 21:32:09) [GCC

    4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import html >>> import requests >>> page = requests.get("http://localhost:8888/index.html") >>> tree = html.fromstring(page.content) >>> print(tree) <Element html at 0x10eaf7d68> >>> print(tree.tag) html >>> len(tree) 2 >>> tree[0] <Element head at 0x10eaf7ef8> >>> tree[1] <Element body at 0x10eaf7db8> >>> for node in tree: ... print(node) ... <Element head at 0x10eaf7ef8> <Element body at 0x10eaf7db8> html head body p div span ul li li li title img
  19. >>> tree[0] <Element head at 0x10eaf7ef8> >>> tree[1] <Element body

    at 0x10eaf7db8> >>> body = tree[1] >>> len(body) 2 >>> body[0] <Element p at 0x10eaf7d68> >>> body[1] <Element div at 0x10eaf7ef8> >>> for node in body: ... print(node) ... <Element p at 0x10eaf7d68> <Element div at 0x10eaf7ef8> html head body p div span ul li li li title img
  20. >>> body = tree[1] >>> len(body) 2 >>> body[0] <Element

    p at 0x10eaf7d68> >>> body[1] <Element div at 0x10eaf7ef8> >>> for node in body: ... print(node) ... <Element p at 0x10eaf7d68> <Element div at 0x10eaf7ef8> >>> for node in body: ... print(node) ... for subnode in node: ... print(' ',subnode) ... <Element p at 0x10eaf7d68> <Element div at 0x10eaf7ef8> <Element span at 0x10eaf7d68> <Element ul at 0x10eaf7d18> <Element img at 0x10eaf7ea8> html head body p div span ul li li li title img
  21. >>> print(tree) <Element html at 0x10eaf7ef8> >>> tree.xpath('body') [<Element body

    at 0x10eaf7d68>] >>> tree.xpath('head/title')[0] <Element title at 0x10eaf7d68> >>> tree.xpath('head') [<Element head at 0x10eaf7d18>] >>> tree.xpath('head/title')[0].text 'Minha Pagina' >>> tree.xpath('head/title') [<Element title at 0x10eaf7d68>] >>> tree.xpath('body/div/ul') [<Element ul at 0x10eaf7d68>] >>> tree.xpath('body/*/ul') [<Element ul at 0x10eaf7d68>] html head body p div span ul li li li title img >>> tree.xpath('//ul') [<Element ul at 0x10eaf7d68>]
  22. >>> tree.xpath('body/div/ul/li') [<Element li at 0x10eaf7f98>, <Element li at 0x10eaf7ea8>,

    <Element li at 0x10eb120e8>] >>> tree.xpath('body/div/ul/li')[1] <Element li at 0x10eb120e8> >>> tree.xpath('//li') [<Element li at 0x10eaf7ea8>, <Element li at 0x10eaf7f98>, <Element li at 0x10eb120e8>] >>> tree.xpath('body/div/ul/li')[1].text 'Aleluia' >>> tree.xpath('//li')[1].text 'Aleluia' >>> tree.xpath('*/li') [] html head body p div span ul li li li title img
  23. >>> tree.xpath('//img') [<Element img at 0x10eb12098>] >>> tree.xpath('//img')[0].text >>> >>>

    tree.xpath("//@src='https://cdn4.iconfinder.com/data/icons/reaction/32/happy-128.png'") True >>> tree.xpath('//img')[0].get('src') 'https://cdn4.iconfinder.com/data/icons/reaction/32/happy-128.png' >>> tree.xpath("//img[@src]") [<Element img at 0x10eb12098>] html header body p div span ul li li li title img >>> tree.xpath('//@src') ['https://cdn4.iconfinder.com/data/icons/reaction/32/happy-128.png'] http://www.w3schools.com/xsl/xpath_syntax.asp