CloudML Talk at DevFest, Madurai

Contains details on CloudML, Big Data and Cloud API

Transcript

1. • BigData • BigQuery • CloudML • Cloud API
2. Karthik Padmanabhan Developer Relations @ karthik_padman
3. Big data and machine learning at Google Big Query Cloud Dataflow Cloud ML Anything you can ask in SQL Parallel processing, batch and stream Machine learning, neural networks
4. Big data and machine learning at Google Big Query Cloud Dataflow Cloud ML Anything you can ask in SQL Parallel processing, batch and stream Machine learning, neural networks
5. Big data and machine learning at Google Big Query Cloud Dataflow Cloud ML Apache Beam Tensorflow Open source
6. Big data and machine learning at Google Big Query Cloud Dataflow Cloud ML Apache Beam Tensorflow Vision API Speech API Translate API Pre-trained models
7. Cloud Dataflow demo
8. Vision API demo
9. Tensorflow demo
10. Photo credit: Matt Chan photo credit - isaiah115 on flickr
11. Photo credit: Matt Chan
12. Google Research Publications
13. Google Research Publications
14. Open Source Implementations Bigtable Flume Dremel
15. Managed Cloud Versions Bigtable Bigtable Flume Dataflow Dremel BigQuery
16. BigQuery demo
17. Google BigQuery Google BigQuery
18. 02 Count some stuff
19. SELECT count(word) FROM publicdata:samples.shakespeare Words in Shakespeare
20. SELECT sum(requests) as total FROM [fh-bigquery:wikipedia.pagecounts_20150212_01] Wikipedia hits over 1 hour
21. SELECT sum(requests) as total FROM [fh-bigquery:wikipedia.pagecounts_201505] Wikipedia hits over 1 month
22. Several years of Wikipedia data SELECT sum(requests) as total FROM [fh-bigquery:wikipedia.pagecounts_201105], [fh-bigquery:wikipedia.pagecounts_201106], [fh-bigquery:wikipedia.pagecounts_201107], ...
23. SELECT SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")') Several years of Wikipedia data
24. How about a RegExp SELECT SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_2015[0-9]{2}$")') WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))
25. 03 How did it do that? o_O
26. Qualities of a good RDBMS
27. Qualities of a good RDBMS • Inserts & locking • Indexing • Cache • Query planning
28. Qualities of a good RDBMS • Inserts & locking • Indexing • Cache • Query planning
32. Storing data -- -- -- -- -- -- -- -- -- -- -- -- Table Columns Disks
33. Reading data: Life of a BigQuery SELECT sum(requests) as sum FROM ( SELECT requests, title FROM [fh-bigquery:wikipedia.pagecounts_201501] WHERE (REGEXP_MATCH(title, '[Jj]en.+')) )
34. Life of a BigQuery L L M Mixer Leaf Storage
35. L L L L M M M Life of a BigQuery Root Mixer Mixer Leaf Storage
36. Life of a BigQuery Query L L L L M M M Root Mixer Mixer Leaf Storage
37. Life of a BigQuery Life of a BigQuery Root Mixer Mixer Leaf Storage SELECT requests, title L L L L M M M
38. Life of a BigQuery Life of a BigQuery Root Mixer Mixer Leaf Storage 5.4 Bil SELECT requests, title WHERE (REGEXP_MATCH(title, '[Jj]en.+')) L L L L M M M
39. Life of a BigQuery Life of a BigQuery Root Mixer Mixer Leaf Storage 5.4 Bil SELECT sum(requests) 5.8 Mil WHERE (REGEXP_MATCH(title, '[Jj]en.+')) SELECT requests, title L L L L M M M
40. Life of a BigQuery Life of a BigQuery Root Mixer Mixer Leaf Storage 5.4 Bil SELECT sum(requests) 5.8 Mil WHERE (REGEXP_MATCH(title, '[Jj]en.+')) SELECT requests, title SELECT sum(requests) L L L L M M M
41. 04 Something Useful Use Wikipedia data to pick a movie
42. 1. Wikipedia edits 2. ??? 3. Movie recommendation
43. Follow the edits
44. Follow the edits Same editor
45. select title, id, count(id) as edits from [publicdata:samples.wikipedia] where title contains 'Hackers' and title contains '(film)' and wp_namespace = 0 group by title, id order by edits limit 10 Pick a great movie
46. select title, id, count(id) as edits from [publicdata:samples.wikipedia] where contributor_id in ( select contributor_id from [publicdata:samples.wikipedia] where id=264176 and contributor_id is not null and is_bot is null and wp_namespace = 0 and title CONTAINS '(film)' group by contributor_id) and wp_namespace = 0 and id != 264176 and title CONTAINS '(film)' group each by title, id order by edits desc limit 100 Find edits in common
47. Discover the most broadly popular films select id from ( select id, count(id) as edits from [publicdata:samples.wikipedia] where wp_namespace = 0 and title CONTAINS '(film)' group each by id order by edits desc limit 20)
48. Edits in common, minus broadly popular select title, id, count(id) as edits from [publicdata:samples.wikipedia] where contributor_id in ( select contributor_id from [publicdata:samples.wikipedia] where id=264176 and contributor_id is not null and is_bot is null and wp_namespace = 0 and title CONTAINS '(film)' group by contributor_id) and wp_namespace = 0 and id != 264176 and title CONTAINS '(film)' and id not in ( select id from ( select id, count(id) as edits from [publicdata:samples.wikipedia] where wp_namespace = 0 and title CONTAINS '(film)' group each by id order by edits desc limit 20 ) ) group each by title, id order by edits desc limit 100
49. Interesting challenges await
50. The plan 01 02 03 04 05 A (very) brief overview of machine learning Vision API Speech API Natural Language API Tears (of joy)
51. Confidential & Proprietary Google Cloud Platform 51 Machine Learning is using many examples to answer questions
52. Confidential & Proprietary Google Cloud Platform 52
53. Confidential & Proprietary Google Cloud Platform 53 Why the sudden explosion in machine learning?
54. Confidential & Proprietary Google Cloud Platform 54
55. Confidential & Proprietary Google Cloud Platform 55
56. Confidential & Proprietary Google Cloud Platform 56
57. Confidential & Proprietary Google Cloud Platform 57 Google Cloud is The Datacenter as a Computer
58. Confidential & Proprietary Google Cloud Platform 58
59. Confidential & Proprietary Google Cloud Platform 59
60. Confidential & Proprietary Google Cloud Platform 60
61. Confidential & Proprietary Google Cloud Platform 61
62. Confidential & Proprietary Google Cloud Platform 62 So what's special? • Sound → Text • Pixels → Meaning Understanding the real world is hard
63. Confidential & Proprietary Google Cloud Platform 63 How can we make it easier?
64. Confidential & Proprietary Google Cloud Platform 64 Cloud Speech API Cloud Vision API
65. Confidential & Proprietary Google Cloud Platform 65 65 Speech API • Speech to text transcription in over 80 languages • Supports streaming and non-streaming recognition • Filters inappropriate content • Demo!
66. Confidential & Proprietary Google Cloud Platform 66 66 Vision API • Label • Landmark • Logo • Face • Text • Safe search Photo attributions: Eiffel Tower (Creative Commons via Sathish J), Lens (Creative Commons via Mark Hunter)
67. 67 { "labelAnnotations": [ { "mid": "/m/0c9ph5", "description": "Flower", "score": 98 }, { "mid": "/m/05s2s", "description": "Plant", "score": 93 }, { "mid": "/m/03bmqb", "description": "Flora", "score": 83 }, { "mid": "/m/0k3b9", "description": "Hydrangea", "score": 81 }, ] } Label Detection 67
68. 68 { "landmarkAnnotations" : [ { "boundingPoly" : { "vertices" : [ { "x" : 52, "y" : 25 }, ... ] }, "mid" : "\/m\/0b__kbm", "score" : 0.4231607, "description" : "The Wizarding World of Harry Potter", "locations" : [ { "latLng" : { "longitude" : -81.471261, "latitude" : 28.473 } } ] } ] } Landmark Detection 68
69. 69 { ... "itemListElement": [ { "@type": "EntitySearchResult", "result": { "@id": "kg:/m/0b__kbm", "name": "The Wizarding World of Harry Potter", ... "detailedDescription": { "articleBody": "The Wizarding World of Harry Potter is a themed area spanning two theme parks – Islands of Adventure and Universal Studios Florida – at the Universal Orlando Resort in Orlando, Florida, USA.\n", ... Knowledge Graph sidebar GET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}
70. 70 "faceAnnotations" : [ { "headwearLikelihood" : "VERY_UNLIKELY", "surpriseLikelihood" : "VERY_UNLIKELY", "rollAngle" : 8.5484314, "angerLikelihood" : "VERY_UNLIKELY", "detectionConfidence" : 0.9996134, "joyLikelihood" : "VERY_LIKELY", "panAngle" : 18.178885, "sorrowLikelihood" : "VERY_UNLIKELY", "tiltAngle" : -12.244568, "underExposedLikelihood" : "VERY_UNLIKELY", "blurredLikelihood" : "VERY_UNLIKELY" "landmarks" : [ { "type" : "LEFT_EYE", "position" : { "x" : 268.25815, "y" : 491.55255, "z" : -0.0022390306 } }, ... Face Detection 70 { "type" : "RIGHT_EYE", "position" : { "x" : 418.42868, "y" : 508.22632, "z" : 49.302765 } }, { "type" : "MIDPOINT_BETWEEN_EYES", "position" : { "x" : 359.86551, "y" : 500.2868, "z" : -7.9241152 } }, { "type" : "NOSE_TIP", "position" : { "x" : 358.51404, "y" : 611.80286, "z" : -31.350466 } }, ...
71. Confidential & Proprietary Google Cloud Platform 71
72. Confidential & Proprietary Google Cloud Platform 72
73. Confidential & Proprietary Google Cloud Platform 73
74. Confidential & Proprietary Google Cloud Platform 74 How about some meaning in those words?
75. Confidential & Proprietary Google Cloud Platform 75 Natural Language API Three methods: 1. Analyze entities - Montreal is a city in Canada 2. Analyze sentiment - I love Montreal 3. Analyze syntax - Michelle Obama is married to Barack Obama
76. Confidential & Proprietary Google Cloud Platform 76 https://cloud.google.com/nl
77. 77 Free tears!
78. 78 • Vision API - 1,000 requests / month • Speech API - 60 minutes / month • Natural Language API - 5,000 units / month (1 unit = 1000 unicode characters) Free tears! tiers
79. Thank you! @karthik_pad man Resources: Speech API cloud.google.com/speech Vision API cloud.google.com/vision Natural Language API cloud.google.com/nl
80. Thank you! Karthik Padmanabhan Developer Relations Google Cloud Platform @ karthik_padman Try BigQuery: bigquery.google.com Cloud API CloudML Slides:
82. About you • Game developers? • Data people? • Students? • Not techies at all?