Scraping (some of) the NYPL Catalog: A story of Requests and BeautifulSoup

An introduction to web scraping in Python.
Pygotham, August 16th, 2015

Transcript

1. Scraping  (some  of)  the  NYPL   Catalog   A  story  of  Requests  and   Beau@fulSoup   Susan  Steinman     Pygotham   August  16th,  2015  
2. •  What  web  scraping  is   •  Why  I  wanted  to  do  it   •  How  it  can  be  done  in  Python   •  What  to  do  with  the  data  you  scrape   •  Hands  on!  
3. Scraping  
4. The  Project  
5. The  Project  
6. The  Project  
7. The  Project  
8. My  code   Catalog   Hello?
9. My  code   Catalog   Hello? Catalog’s  Code  
10. My  code   Catalog’s  Code   API?  
11. My  code   Catalog   Web  Scraping  
12. Catalog   Inspect!  
13. Inspect!  
14. Inspect!   HTML
15. Let’s  inspect  together   •  Find  a  friend!   •  Pick  a  site.   •  Inspect       hUp://on.nypl.org/1OTIe9i    
16. How?   HTML My  code  
17. Make  a  request   HTML My  code  
18. The  Requests  Module   >>>  r=requests.get(“[URL]”)   >>>  print  r   <Response  [200]>   >>>  print  r.content  
19. Let’s  Request  Together   >>>  pip  install  requests   •  Or  ask  a  friend  
20. Let’s  Request  Together   •  Open  your  terminal   •  Type  in  “Python”  to  work  in  your  REPL   •  Import  requests   •  Request  the  site!   hUp://on.nypl.org/1OTIe9i    
21. What  did  you  find?  
23. $  python   >>>  import  requests   >>>  r=requests.get("hUps:// nypl.bibliocommons.com/item/show_circula@on/ 20029058052907")   >>>  print  r   <Response  [200]>    
24. >>>  print  r.content            <div  class="back_container">          <a  href="/item/show/20029058052907"   class="back_to">&laquo;  Back  to  Mockingjay</ a>              </div>             <div  class="content">      <div  id="availabilityInfo_665580990"   tes@d="availability_info_modal">   <a  name="modaltop"></a>        <div  class="bib_@tle"  tes@d="availability_@tle">            
25. >>>  type(r.content)   <type  ‘str’>   >>>  
27. HTML  Parser  
29. <tr  class="odd”>      <td  tes@d="item_branch_name"  scope="row">        115th  Street    (4)</td>        <td  tes@d="item_collec@on">  115th  Street  YA        Fic@on  </td>        <td  tes@d="item_call_number">  FIC  C    </td>        <td  tes@d="item_status”>  Available  </td>            <td  class="date"  tes@d="item_due_date">    </td>   </tr>  
30. Beau@fulSoup4   >>>  from  bs4  import  Beau@fulSoup   >>>  soup  =  Beau@fulSoup(r.content)   >>>  row  =  soup.find(“tr”)      
31. <tr  class="odd”>      <td  tes@d="item_branch_name"  scope="row">        115th  Street    (4)</td>        <td  tes@d="item_collec@on">  115th  Street  YA        Fic@on  </td>        <td  tes@d="item_call_number">  FIC  C    </td>        <td  tes@d="item_status”>  Available  </td>            <td  class="date"  tes@d="item_due_date">    </td>   </tr>  
32. Beau@fulSoup4   >>>  from  bs4  import  Beau@fulSoup   >>>  soup  =  Beau@fulSoup(r.content)   >>>  row  =  soup.find(“tr”)   >>>  name  =  row.find(tes@d="item_branch_name”)   >>>  print  name.text                                115th  Street      
33. mapmybook.herokuapp.com  
34.   github.com/susinmo@on/mapmybook2   github.com/susinmo@on/NYPL_API        
35. Ques@ons?                      susan.steinman@gmail.com