Waldo-gcp

Google I/O 2015 Extend Taipei - Travel Service at Google Cloud Platform

Transcript

1. Travel Service at GCP waldo-gcp Cage, F2E, Mitac
2. Road trip stopping at popular U.S. cities - Computing the optimal road trip across the U. S. | Randal S. Olson(http://goo.gl/pDNfp0)
3. My own road trip (https://goo.gl/SzBwbM)
4. Waldo-gcp overview What is waldo
5. Demo https://waldo-gcp.appspot.com/
6. Waypoints address sample text file What is waldo "台北市內湖區瑞光路227號1樓", "高雄市鼓山區美術東二路106號", "台南市長榮路一段175號", "臺北市松山區南京東路五段123巷1弄15號", "高雄市五福四路131號2樓"
7. Waldo-gcp Google Cloud Platform Features What is waldo-gcp Compute Storage App Engine Cloud Datastore Cloud Storage Compute Engine
8. Waldo-gcp architecture queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot RESTful APIs for waypoints list, upload waypoints optim al road trip
9. Waldo-gcp dispatch.yaml dispatch: - url: "*/favicon.ico" module: default - url: "waldo-gcp.appspot.com/" module: default - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/google40fc41ccc276e5c5.html" module: ownership - url: "*/*" module: "default" Waldo-gcp architecture Application hierarchy [App Engine Modules in Python](https://goo.gl/p3nl48)
10. Compute
11. Cloud Computing Compute Paas IaaS Application Data Runtime Middleware OS Virtualization Servers Storage Networking Application Data Runtime Middleware OS Virtualization Servers Storage Networking Saas Application Data Runtime Middleware OS Virtualization Servers Storage Networking Infrastructure as a Service Platform as a Service Service as a Service You manage Google manage
12. Managed VMs Compute | Managed VMs • Managed VMs ◦ The Flexibility of Compute Engine ◦ The productivity of App Engine • Provides best of both ◦ IaaS + PaaS
13. Waldo-gcp storage - cloud datastore queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
14. Waldo-gcp storage - cloud datastore queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
15. Storage
16. Waldo-gcp Storage Storage Cloud Datastore Cloud Storage Cloud SQL Compute Engine (Persistent Disk) Blob NoSQL SQL Block
17. Waldo-gcp storage - cloud datastore queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
18. Waldo-gcp storage - cloud storage queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
19. Waldo-gcp storage - cloud storage queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
20. Setup Cloud Storage Buckets
21. Waldo-gcp cloud storage queue Metadata Waldo Server App Engine Task Queues Cloud Storage Optimal- waypoints managed VMs (python27) ... endpoints API (OAuth 2) Object Notification Cloude Datastore snapshot managed VMs (nodejs) Android App CSV upload waypoints Store/retrieve metadata waypoint map snapshot optim al road trip RESTful APIs for waypoints list, upload waypoints
22. Waypoints GCS Bucket Waldo-gcp cloud storage Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image endpoint API upload waypoints address to managed VMs upload optimal address to managed VMs upload screenshot image to
23. GCS Bucket Notification Waldo-gcp setup gcs object notification Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image
24. GCS Bucket Notification Waldo-gcp storage Waldo Server Android App endpoints API (OAuth 2) Waldo-gcp-file Waldo-gcp-file-in Waldo-gcp-file-image • bucket push notification ◦ https only ◦ domain owned /notice
25. Google Cloud Storage Notification class BucketNotificationHandler(webapp2.RequestHandler): def post(self): # Decode the name and bucket of the notification def notification_decoder(object): if 'kind' in object: return dict(name=object['name'], bucket=object['bucket'], id=object['id']) ... # Add a new task to the queue taskqueue.add(url='/waypoints/', params={ 'bucket': notification['bucket'], 'name': notification['name'], 'id': notification['id'] },queue_name='new-files') ... Create the App Engine backend
26. Waypoints address text file Create the App Engine backend | Waldo-gcp-file "台北市內湖區瑞光路227號1樓", "高雄市鼓山區美術東二路106號", "台南市長榮路一段175號", "臺北市松山區南京東路五段123巷1弄15號", "高雄市五福四路131號2樓"
27. Waypoints address distance_m & distance_s Create the App Engine backend | Waldo-gcp-file-in waypoint1 waypoint2 distance_m duration_s 台南市長榮路一段175號 高雄市五福四路131號2樓 89300 4375 台南市長榮路一段175號 高雄市鼓山區美術東二路 106號 84212 4243 台北市內湖區瑞光路227號1樓 臺北市松山區南京東路五段 123巷1弄15號 5204 763 高雄市五福四路131號2樓 高雄市鼓山區美術東二路 106號 4163 732 台北市內湖區瑞光路227號1樓 高雄市五福四路131號2樓 356184 13972 臺北市松山區南京東路五段 123巷1弄15號 高雄市鼓山區美術東二路 106號 354392 14071 臺北市松山區南京東路五段 123巷1弄15號 高雄市五福四路131號2樓 359450 14230 台北市內湖區瑞光路227號1樓 台南市長榮路一段175號 275002 10503 台北市內湖區瑞光路227號1樓 高雄市鼓山區美術東二路 106號 352764 13541 臺北市松山區南京東路五段 123巷1弄15號 台南市長榮路一段175號 276930 10674
28. Create Managed VMs Server • Optimal road trip • screenshot
29. Waldo-gcp waypoint optimization • Gather the distance between wall waypoints • Google Map APIs • genetic algorithm optimal road trip [nbviewer.ipython.org/github/rhiever/Data-Analysis-and-Machine-Learning- Projects/blob/master/optimal-road-trip/Computing the optimal road trip across the U.S..ipynb](http://goo.gl/SYoh53)
30. Pandas • Python library • Powerful data structures for data analysis, time series,and statistics optimal road trip http://pandas.pydata.org/
31. Using Managed VMs Waldo-gcp architecture Waldo Server Android App endpoints API (OAuth 2) /notice - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/*" module: default bucket notification entry point optimal road trip screenshot waldo-gcp
32. Managed VMs Waldo-gcp architecture Waldo Server Android App endpoints API (OAuth 2) /notice - url: "*/notice/*" module: notice - url: "*/waypoints/*" module: optimal-waypoints - url: "*/screenshot/*" module: screenshot - url: "*/*" module: default Auto-scaled managed VMs (python27) ... managed VMs (nodejs)
33. Containers on Managed VMs FROM google/appengine-python27 # Install pandas RUN apt-get update && apt-get install -y -q --no-install- recommends python-pandas python-numpy # Install Google Map APIs, Google Client Library ADD requirements.txt /app/ RUN pip install --upgrade pip RUN pip install -r requirements.txt ADD . /app Create the App Engine | Dockerfile [Managed VMs - App Engine — Google Cloud Platform](https://goo.gl/4FFSmm)
34. Optimal waypoint Module Configuration application: waldo-gcp module: optimal-waypoints version: 1 runtime: python27 api_version: 1 threadsafe: yes vm: true automatic_scaling: min_num_instances: 1 max_num_instances: 5 cool_down_period_sec: 60 cpu_utilization: target_utilization: 0.5 Create the App Engine | app.yaml
35. Waldo-gcp screenshot • Screenshot as a service • Phantomjs + nodejs snapshot server [fzaninotto/screenshot-as-a-service](https://github.com/fzaninotto/screenshot-as- a-service)
36. Screenshot Module Configuration application: waldo-gcp module: screenshot version: 1 runtime: custome api_version: 1 threadsafe: yes vm: true resources: cpu: .5 memory_gb: 1.3 manual_scaling: instances: 1 snapshot server
37. Waldo-gcp
38. Road trip on Google Map waldo-gcp google map optimal_route = [/*optimal waypoint calculate by genetic algorithm*/] optimal_route = list(optimal_route) optimal_route += [optimal_route[0]] subset = 0 while subset < len(optimal_route): waypoint_subset = optimal_route[subset:subset + 10] output = "calcRoute(\"%s\", \"%s\", [" % (waypoint_subset[0], waypoint_subset[-1]) for waypoint in waypoint_subset[1:-1]: output += "\"%s\", " % (waypoint) if len(waypoint_subset[1:-1]) > 0: output = output[:-2] output += "]);" print(output) print("") subset += 9 ... • Google Map APIs ◦ split max waypoints (11) in one route
39. Road trip on Google Map waldo-gcp google map calcRoute("605嘉義縣阿里山 鄉二萬坪106號", "605嘉義縣阿里山 鄉二萬坪106號", ["500彰化縣彰化市溫泉 路31號", "224九份風景區瑞芳區新北市 ", "971花蓮縣新城 鄉海岸路", "555南投縣魚池 鄉中山路599號"]); function calcRoute(start, end, routes) { switch (start) { case "605嘉義縣阿里山 鄉二萬坪106號" ,"605嘉義縣阿里山 鄉二萬坪106號": directionsDisplay1 = new google.maps.DirectionsRenderer(directionsDisplayOptions); .. } var request = {origin: start, destination: end, waypoints: waypts, optimizeWaypoints: false, travelMode: google.maps.TravelMode.DRIVING }; directionsService.route(request, function (response, status) { case "605嘉義縣阿里山 鄉二萬坪106號" ,"605嘉義縣阿里山 鄉二萬坪106號": directionsDisplay1.setDirections(response); } }); .. } } 1. 605嘉義縣阿里山鄉二萬坪106號 2. 500彰化縣彰化市溫泉路 31號 3. 224九份風景區瑞芳區新北市 4. 971花蓮縣新城鄉海岸路 5. 555南投縣魚池鄉中山路599號
40. Thanks you!
41. • [Here's Waldo: Computing the optimal search strategy for finding Waldo | Randal S. Olson](http://goo. gl/DSOZbv) • [Computing the optimal road trip across the U.S. | Randal S. Olson](http://goo.gl/pDNfp0) • [nbviewer.ipython.org/github/rhiever/Data-Analysis-and-Machine-Learning-Projects/blob/master/optimal- road-trip/Computing the optimal road trip across the U.S..ipynb](http://goo.gl/SYoh53) • [GoogleCloudPlatform/abelana-gcp](https://goo.gl/6L55oa) Reference
43. 《威利在哪裡？》（英文：Where's Wally?），也譯《威力在哪裡？》、《大家一起找沃裡？》、《聰明的沃 裡？》，是一套由英國插畫家Martin Handford創作的兒童書籍。這個書的目標就是在一張人山人 海的圖片中找出一個特定的人物 ——威利。威利穿著紅白條紋的襯衫並戴著一個絨球帽，手上拿 著木製的手杖，還戴著一副眼鏡。他總是會弄丟東西，如書本、野營設備甚至是他的鞋子，而讀者 也要幫他在圖中找出這些東西來。 wikipedia http://goo.gl/D4mH6 Where's Waldo?