Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DroidKaigi 2020 - Interactive Canvasを使ってGUIを持ったActions on Googleを作る

DroidKaigi 2020 - Interactive Canvasを使ってGUIを持ったActions on Googleを作る

DroidKaigi 2020 - Interactive Canvasを使ってGUIを持ったActions on Googleを作る(https://droidkaigi.jp/2020/timetable/156644
の登壇用資料です。
残念ながら、Covid19の影響でDroidKaigiが中止となり、登壇はできませんでしたが、録画のセッションとなったこちらの動画でこの資料を使っています。
https://www.youtube.com/watch?v=XdVH4YMlNxw

F3b8da330ac8f07f3aa4d8e240506b6b?s=128

Takamitsu Mizutori

December 03, 2020
Tweet

Transcript

  1. *OUFSBDUJWF$BOWBTΛ࢖ͬͯ(6*Λ࣋ͬͨ "DUJPOTPO(PPHMFΛ࡞Δ !NJ[VUPSZ ਫௗܟຬ

  2. ࣗݾ঺հ
 ├── ਫௗܟຬ @mizutory ├── Goldrush Computing୅ද └── ϓϩάϥϚʔ ├──

    Android ├── iOS ├── Actions On Google ├── Vue.js └── Python
  3. "DUJPOͱ͸ (PPHMF/FTU)VC (PPHMF)PNF ͋Δ͍͸"OESPJEεϚʔτϑΥϯͷ (PPHMF"TTJTUBOUͰಈ࡞͢ΔԻ੠Ͱͷର࿩ͰϢʔβʔͱ΍ΓͱΓΛ͢ΔΞϓϦ ͷ͜ͱΛʮΞΫγϣϯʯ͋Δ͍͸ʮ"DUJPOTPO(PPHMFʯͱݴ͍·͢ɻ

  4. "DUJPOಈ࡞·Ͱͷ࢓૊Έ "DUJPO w "TTJTUBOU4QFFDIUP5FYU w %JBMPH'MPXςΩετʹͳͬͨձ࿩ͷड͚ࡼɺΩʔϫʔυநग़ w 8FCIPPLձ࿩ͷड͚ࡼ͝ͱʹɺฦ౴ͷ࡞੒

  5. *OUFOU *OUFOUɿ̍ͭͷձ࿩ͷड͚ࡼ 5SBJOJOHQISBTFTϢʔβʔ͕஻ͬͯ͘ΔͰ͋Ζ͏ձ࿩ͷ૝ఆͰ͢ɻ

  6. &OUJUZ &OUJUZநग़͍ͨ͠Ωʔϫʔυͷఆٛू

  7. *OUFOUͱ&OUJUZͰձ࿩Λड͚Δ

  8. 8FCIPPLड͚ͨձ࿩ΛIPPLͯ͠ฦ౴Λੜ੒

  9. 8FCIPPL'VMpMMNFOU'JSFCBTF'VODUJPOTˠฦ౴Λੜ੒͢Δ8FC"1* *OUFOUˠ̍ͭ̍ͭͷձ࿩ͷड͚ࡼ &OUJUZˠநग़͍ͨ͠Ωʔϫʔυͷఆٛू ʢ8FCIPPLɿ8FCଆͰϦΫΤετΛҾֻ͚ͬΔʣ ʢ'VMMpMMNFOUˠฦ౴ΛpMMͯ͋͛͠Δ΋ͷ

  10. 'JSFCBTF'VODUJPOTͷ࡞Γํ $ firebase init Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook

  11. parrot ├── firebase.json └── functions ├── index.js ├── node_modules ├──

    package-lock.json └── package.json *OUFOUϋϯυϥʔ
  12. 'use strict'; // Actions on Google client library͔ΒɺDialogflowϞδϡʔϧΛΠϯϙʔτ͢Δ const {dialogflow}

    = require('actions-on-google'); // firebase-functions packageΛΠϯϙʔτ͢Δ const functions = require('firebase-functions'); // Dialogflow clientΠϯελϯεΛ࡞Δ const app = dialogflow({debug: true}); // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, fish, any}) => { if (animal) { conv.close(animal + '͸ಈ෺Ͱ͢Ͷ'); }else if (fish) { conv.close(fish + '͸ڕͰ͢Ͷ'); }else { conv.close(any + '͸ಈ෺Ͱ΋ڕͰ΋͋Γ·ͤΜ'); } }); // DialogflowApp ΦϒδΣΫτΛHTTPS POSTϦΫΤετͷϋϯυϥʔͱͯ͠ొ࿥ exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app); *OUFOUϋϯυϥʔͷྫ *OUFOU໊ ύϥϝʔλʔ໊ʢ&OUJUZ໊ʣ *OUFOUˠɹձ࿩ͷड͚ࡼɹˠɹϋϯυϥʔ &OUJUZˠɹநग़͍ͨ͠Ωʔϫʔυɹˠɹύϥϝʔλʔ
  13. ಈ෺൛ࢁखઢήʔϜ ಈ෺͠ΓͱΓ *OUFSBDUJWF$BOWBT

  14. *OUFOUͷ४උ

  15. *OUFSBDUJWF$BOWBTΛߟྀͨ͠ 'JSFCBTF'VODUJPOTͷ࡞Γํ $ firebase init Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ

    -> Interactive Canvas
  16. parrot ├── firebase.json ├── functions │ ├── index.js │ ├──

    node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html └── index.html functionsϑΥϧμ → Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook publicϑΥϧμ → Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ -> Interactive Canvas
  17. // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, pass, any}) => { if

    (animal) { var result = animals.indexOf(animal); …ɹաڈʹಡΈ্͛ͨಈ෺͡Όͳ͍͔ͷνΣοΫ … //ࠓ౓͸AssistantͷಡΈ্͛Δಈ෺ΛϥϯμϜʹબͿ let newAnimal = getRandomMember(animals); …ɹաڈʹಡΈ্͛ͨಈ෺͡Όͳ͍͔ͷνΣοΫ … //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); }else { conv.close(any + '͸ಈ෺Ͱ͸͋Γ·ͤΜɻ͋ͳͨͷෛ͚Ͱ͢'); consumedAnimals.length = 0; } }); *OUFSBDUJWF$BOWBT༻ͷϋϯυϥʔ
  18. //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://$ {firebaseConfig.projectId}.firebaseapp.com/`, data: {

    scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); )UNM3FTQPOTFΫϥεͷΦϒδΣΫτΛฦ͢ VSMϑϩϯτΤϯυ JOEFYIUNM ͕ஔ͍ͯ͋Δ৔ॴ EBUBϑϩϯτΤϯυʹૹΔσʔλ
  19. None
  20. 8FCIPPL͔ΒͷϨεϙϯεΛݟͯΈΔ

  21. { "payload": { "google": { "expectUserResponse": true, "richResponse": { "items":

    [ { "htmlResponse": { "url": "https://parrot-9855a.firebaseapp.com/", "updatedState": { "scene": "animal", "name": "γϩΫϚ", "picture": "polarbear.jpg" } } }, { "simpleResponse": { "textToSpeech": "γϩΫϚ" } } ] } } } } (PPHMF/FTU)VCͰ͸͜ΕΛ ड͚औͬͯɺ ΢Σϒϖʔδͷϩʔυͱ ൃ࿩Λߦ͏ 8FCIPPL͔ΒͷϨεϙϯεΛݟͯΈΔ
  22. //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene:

    'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } function hasInteractiveCanvas(conv) { const hasInteractiveCanvas = ɹɹɹɹɹɹɹɹɹ conv.surface.capabilities.has('actions.capability.INTERACTIVE_CANVAS'); return hasInteractiveCanvas } ஫ʣσόΠε͕*OUFSBDUJWF$BOWBTͷػೳΛ࣋ͬͯ ͍Δͱ͖͚ͩɺ)UNM3FTQPOTFΛฦ͢Α͏ʹ͢Δ
  23. parrot ├── firebase.json ├── functions │ ├── index.js │ ├──

    node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html └── index.html functionsϑΥϧμ → Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook publicϑΥϧμ → Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ -> Interactive Canvas ͕͜͜ϩʔυ͞ΕΔ 8FCϖʔδ
  24. parrot ├── firebase.json ├── functions │ ├── animal.js │ ├──

    index.js │ ├── node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html ├── css │ └── index.css ├── images │ ├── africaboar.jpg │ ├── africaelephant.jpg │ ├── akagitsune.jpg │ ├── alpaca.jpg │ └── zou.jpg ├── index.html └── js └── index.js 'JSFCBTF)PTUJOH޲͚ϑ ϩϯτΤϯυϓϩδΣΫτ ͷߏ଄
  25. JOEFYIUNM <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <meta name="viewport"

    content="width=device-width, initial-scale=1" /> <title>[؆қ൛]ಈ෺͠ΓͱΓ</title> <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;," /> <link rel="stylesheet" href="css/index.css" /> <script src="https://www.gstatic.com/assistant/interactivecanvas/api/ interactive_canvas.min.js"></script> </head> <body> <div class="container"> <div id="animal"> <div id="image_block"> <img id="animalimage" /> </div> <div id="name"> </div> </div> </div> <script src="js/index.js"></script> </body> </html>
  26. JOEFYKT window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data)

    } }) } function onDataReceived(data){ if (data.scene === 'animal') { if(data.picture){ document.querySelector('#animalimage').src = `images/${data.picture}`; }else{ document.querySelector('#animalimage').src = `images/no_image.jpg`; } document.querySelector('#name').innerText = `${data.name}`; } } JOUFSBDUJWF$BOWBTʹSFBEZΠϕϯ τʹϦεφʔΛొ࿥͠ɺPO6QEBUF ίʔϧόοΫΛొ࿥͢Δ
  27. None
  28. σόοά

  29. σόοάɿ'JSFCBTF)PTUJOH ϩʔΧϧͰͷϨΠΞ΢τͷݕূ ˠQVCMJDJOEFYIUNMΛ։͘ μϛʔσʔλΛೖΕͯͷݕূ σΟϓϩΠޙ͸ɺ IUUQTϓϩδΣΫτ*%pSFCBTFBQQDPN Λ։͍ͯ֬ೝ

  30. μϛʔσʔλΛ࢖ͬͯͷݕূJOEFYKT window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data)

    } }) //Debug༻ onDataReceived({ scene: 'animal', picture: "alpaca.jpg", name: "ΞϧύΧ" }) } function onDataReceived(data){ if (data.scene === 'animal') { if(data.picture){ document.querySelector('#animalimage').src = `images/${data.picture}`; }else{ document.querySelector('#animalimage').src = `images/no_image.jpg`; } document.querySelector('#name').innerText = `${data.name}`; } } ϑϩϯτΤϯυίʔυͷσόοά
  31. σόοάɿ'JSFCBTF'VODUJPOT "DUJPOT0O(PPHMFγϛϡʔϨʔλʔͷ &3303%&#6(3&410/4&λϒΛݟΔ 'JSFCBTF$POTPMF 'VODUJPOTͷϩάΛݟΔ

  32. None
  33. "DUJPOTPO(PPHMF 4JNVMBUPS  &33034 %&#6( 3&410/4&

  34. 'JSFCBTF$POTPMF ˠ'VODUJPOT ˠϩά

  35. ݁߹෦෼ͷςετ ˠ"DUJPOTPO(PPHMF4JNVMBUPSΛσΟϕϩούʔπʔϧͰݟΔ

  36. 5JQT

  37. None
  38. ݩը૾

  39. JOEFYKTɿϔομྖҬͷ֬อ window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data)

    } }) interactiveCanvas.getHeaderHeightPx().then((height) => { document.body.style.paddingTop = `${height}px`; }) }
  40. None
  41. *OUFSBDUJPO

  42. 1BTTϘλϯ

  43. JOEFYIUNM <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <meta name="viewport"

    content="width=device-width, initial-scale=1" /> <title>[؆қ൛]ಈ෺͠ΓͱΓ</title> <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;," /> <link rel="stylesheet" href="css/index.css" /> <script src="https://www.gstatic.com/assistant/interactivecanvas/api/ interactive_canvas.min.js"></script> </head> <body> <div class="container"> <div id="animal"> <div id="image_block"> <img id="animalimage" /> </div> <div id="name"> </div> <button id="pass_button">PASS</button> </div> </div> <script src="js/index.js"></script> </body> </html>
  44. JOEFYKTɿϢʔβʔΠϯλϥΫγϣϯ window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data)

    } }) document.querySelector('#pass_button').addEventListener('click', elem => { interactiveCanvas.sendTextQuery("ύε"); }); interactiveCanvas.getHeaderHeightPx().then((height) => { document.body.style.paddingTop = `${height}px`; }) } zύεzΫΤϦʔͷૹ৴
  45. 1"44ϘλϯΛԡͨ͜͠ͱ͕ʮύεʯ ͱ࿩͔͚ͨ͜͠ͱͱಉ͡ʹͳΔ

  46. // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, pass, any}) => { if

    (animal) { }else if(pass){ //AssistantͷಡΈ্͛Δಈ෺ΛϥϯμϜʹબͿ let newAnimal = getRandomMember(animals); //աڈʹಡΈ্͛ͨΞχϚϧΛௐ΂Δ //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); }else { conv.close(any + '͸ಈ෺Ͱ͸͋Γ·ͤΜɻ͋ͳͨͷෛ͚Ͱ͢'); consumedAnimals.length = 0; } });
  47. None
  48. OPUFDPNNJ[VUPSZ ຊ೔આ໌ͨ͠ϚςϦΞϧ͸ˣˣˣͰ΋ެ։͍ͯ͠·͢ɻ

  49. w "DUJPOTPO(PPHMF w J04ɺ"OESPJEΞϓϦ w 8FCɺϑϩϯτΤϯυʢ7VFKTʣ w 8FC"1*ɺόοΫΤϯυʢOPEFKT 1ZUIPO%KBOHP 

    w ػցֶश w *05ɺϩϘοτ "DUJPOʹؔ͢Δ࣭໰΍ˣˣˣͷ։ൃͷ૬ஊ͸ɺɺ NJ[VUPSJ!HPMESVTIDPNQVUJOHDPN ͪ͜Β·Ͱˠ ʮ%SPJE,BJHJΛݟͨʯͱݴͬͯ΋Β͑Δͱॿ͔Γ·͢ʂ !NJ[VUPSZ