15-437 Google Custom Search

Google Custom Search Thierry Sans

Our web application today Build a web application for searching
SML deﬁnitions from the SML Basis Standard Library ✓ We will use Google custom search for building this app ... because we know you like SML!

The Google APIs

The Google Custom Search API ➡ Use the Google Custom
Search API to search over a website or a collection of websites and to embed the results in your web application

3 ways to use Google Custom search • Google snippet
(client side) • Google Javascript API (client side) • Google Python API (server side)

The Google Snippet

Step 1 - Create a search engine   with Google
Custom Search

Step 2 - Copy the generated code snippet

Step 4 - Paste the code snippet in your page
... <body> <script> (function() { var cx = '009372546338520416759:-rdtwkgor4i'; var gcse = document.createElement('script'); gcse.type = 'text/javascript'; gcse.async = true; gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') + '//www.google.com/cse/cse.js?cx=' + cx; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(gcse, s); })(); </script> <gcse:search></gcse:search> <body></html> search/templates/search/index.html Your Custom Search Identiﬁer  (aka CX or CSEid) The div element where to show the results

Result

Advantages and Limitations ✓ Very easy to use ๏ The
results are shown in a dedicated DOM that is not easy to customize

Solution ➡ On the client side 1. query the Google
search using the Google Javascript API 2. process the results using Javascript 3. show the results in the page

The Google Javascript API

Create a project

Activate the Custom Search API

Get your Google API key from the API Console

Get the Search Engine ID

Search Request function client() { $('#result').html('<script src="https://www.googleapis.com/customsearch/v1?\ key=AIzaSyDBrtHVzXjrvKus_qGpPbTm9rJppIhjxvM&\ cx=004982958264606934300:2o52rkqdfaw&\ q='+$('#inp').val()+'&\
callback=hndlr"></script>'); } function hndlr(response) { .... } search2/static/js/script.js Your API key Your custom search engine id Your search query Your callback method

The callback method ➡ The call method (called hndlr in
the example) deﬁnes what to do with the search results

Example function hndlr(response) { page = “” for (var i
= 0; i < response.items.length; i++) { var item = response.items[i]; var link = item.link; page += "<br/><div><a onclick=\"go('"+link+"');\" href='#'>" +item.htmlTitle +"</a><br/>" +item.htmlSnippet+ + "<br/><a onclick=\"go('"+link+"');\" href='#'>" +link+"</a></div><br/>"; } $('#result').html(page); $('#result').show(); } search2/static/js/script.js For each search ... display the search result but change the target url address so that the link opens in the current page rather than opening a new one

Result

Problem ๏ Nothing happens when you click on the link
➡ No bug shown except ... This is a Cross Domain error

Cross Domain restriction (aka Same Origin Policy) ”The policy permits
scripts running on pages originating from the same site to access each other's methods and properties with no speciﬁc restrictions, but prevents access to most methods and properties across pages on different sites.” Wikipedia

Solution - use an iframe • An <iframe> can contains
another HTML document possibly coming from another domain ➡ Like a webpage inside a webpage ✓ An iFrame is acting like a fence between the two documents <html> . . . <iframe> </iframe> . . . </html> <html> </html> Javascript code from the inner document cannot access the resources from the main document and vice versa

Solution to our problem function hndlr(response) { page = “”
for (var i = 0; i < response.items.length; i++) { var item = response.items[i]; var link = item.link; page += "<br/><div><a onclick=\"go('"+link+"');\" href='#'>" +item.htmlTitle +"</a><br/>" +item.htmlSnippet+ + "<br/><a onclick=\"go('"+link+"');\" href='#'>" +link+"</a></div><br/>"; } $('#result').html(page); $('#result').show(); } search2/static/js/script.js

Result

Advantages and Limitations ✓ Using an iframe is secure ๏
We cannot modify the page contained in the iframe ➡ We want the deﬁne new CSS and Javascript controls for the result document

Solution ➡ On the server side 1. query the Google
search using the Python Google API 2. process the results using Python 3. return the results to the client

The Google Python API

Useful Libraries The Python Google Api  http://code.google.com/p/google-api-python-client/downloads/list ➡ Google APIs
Client Library for Python $ pip install --upgrade google-api-python-client  BeautifulSoup  http://www.crummy.com/software/BeautifulSoup/ ➡ HTML5 parser (and normalizer) $ pip install BeautifulSoup

Example from apiclient.discovery import build from BeautifulSoup import BeautifulSoup import
urllib2 import re @csrf_exempt def find(request): try: service = build("customsearch","v1",\ developerKey="AIzaSyDBrtHVzXjrvKus_qGpPbTm9rJppIhjxvM") res = service.cse().list(q=request.POST['key'],\ cx='004982958264606934300:2o52rkqdfaw').execute() page = urllib2.urlopen(res['items'][0]['link']) soup = BeautifulSoup(page).body except: return HttpResponse("error") else: return HttpResponse(soup) search3/views.py

15-437 Google Custom Search

15-437 Google Custom Search

ThierrySans

More Decks by ThierrySans

Other Decks in Education

Featured

Transcript

Google Custom Search Thierry Sans

Our web application today Build a web application for searching

The Google APIs

The Google Custom Search API ➡ Use the Google Custom

3 ways to use Google Custom search • Google snippet

The Google Snippet

Step 1 - Create a search engine   with Google

Step 2 - Copy the generated code snippet

Step 4 - Paste the code snippet in your page

Result

Advantages and Limitations ✓ Very easy to use ๏ The

Solution ➡ On the client side 1. query the Google

The Google Javascript API

Create a project

Activate the Custom Search API

Get your Google API key from the API Console

Get the Search Engine ID

Search Request function client() { $('#result').html('<script src="https://www.googleapis.com/customsearch/v1?\ key=AIzaSyDBrtHVzXjrvKus_qGpPbTm9rJppIhjxvM&\ cx=004982958264606934300:2o52rkqdfaw&\ q='+$('#inp').val()+'&\

The callback method ➡ The call method (called hndlr in

Example function hndlr(response) { page = “” for (var i

Result

Problem ๏ Nothing happens when you click on the link

Cross Domain restriction (aka Same Origin Policy) ”The policy permits

Solution - use an iframe • An <iframe> can contains

Solution to our problem function hndlr(response) { page = “”

Result

Advantages and Limitations ✓ Using an iframe is secure ๏

Solution ➡ On the server side 1. query the Google

The Google Python API

Useful Libraries The Python Google Api  http://code.google.com/p/google-api-python-client/downloads/list ➡ Google APIs

Example from apiclient.discovery import build from BeautifulSoup import BeautifulSoup import