Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Arcas: Using Python to access open research literature

Nikoleta
August 30, 2017

Arcas: Using Python to access open research literature

EuroScipy 2017

Nikoleta

August 30, 2017
Tweet

More Decks by Nikoleta

Other Decks in Science

Transcript

  1. 0.5min + 100 × 1.5min + 10 × 0.5min =

    155.5min ⇒ 2h and 35.5min
  2. API

  3. API1 Query XML API2 Query XML API3 Query XML API4

    Query XML API5 Query XML API6 Query XML
  4. ARCAS API1 Query XML API2 Query XML API3 Query XML

    API4 Query XML API5 Query XML API6 Query XML
  5. >>> import arcas >>> api = arcas.Arxiv() >>> parameters =

    api.parameters_fix( ... title=’sustainable software’, records=1, start=1) >>> url = api.create_url_search(parameters) >>> request = api.make_request(url) >>> root = api.get_root(request) >>> raw_article = api.parse(root) >>> article = api.to_dataframe(raw_article[0]) >>> api.export(article, "result.json")
  6. {"key":{"0":"Ahern2013"}, "unique_key":{"0":"698d27415f69258ef122f46b184a77e0"}, "title":{"0":"VisIt: Experiences with Sustainable Software"}, "author":{"0":"Sean Ahern","1":"Eric Brugger"},

    "abstract":{"0":" The success of the VisIt visualization..."}, "date":{"0":2013}, "journal":{"0":"arXiv"}, "provenance":{"0":"arXiv"}}
  7. >>> for p in [arcas.Arxiv, arcas.Nature, arcas.Ieee, arcas.Plos]: ... api

    = p() ... parameters = api.parameters_fix( ... title=’sustainable software’, records=1, start=1) ... url = api.create_url_search(parameters) ... request = api.make_request(url) ... root = api.get_root(request) ... raw_article = api.parse(root) ... try: ... for art in raw_article: ... article = api.to_dataframe(art) ... api.export(article, "result_from_{}.json".format( ... api.__class__.__name__)) ... except TypeError: ... pass
  8. 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018

    year 2 4 6 8 10 12 14 16 number of records Articles per Year (N=87)
  9. 2000 2002 2004 2006 2008 2010 2012 2014 2016 year

    0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 number of records Provenance IEEE arXiv PLOS
  10. $ arcas_scrape --version Arcas 0.0.3 $ arcas_scrape -p arxiv -t

    "Sustainable Software" -r 1 http://export.arxiv.org/api/query?search_query=ti:Sustainable Software&max_results=1&start=1