DroidKaigi 2020 - Interactive Canvasを使ってGUIを持ったActions on Googleを作る

DroidKaigi 2020 - Interactive Canvasを使ってGUIを持ったActions on Googleを作る（https://droidkaigi.jp/2020/timetable/156644）
の登壇用資料です。
残念ながら、Covid19の影響でDroidKaigiが中止となり、登壇はできませんでしたが、録画のセッションとなったこちらの動画でこの資料を使っています。
https://www.youtube.com/watch?v=XdVH4YMlNxw

Transcript

1. *OUFSBDUJWF
   $BOWBTΛ࢖ͬͯ(6*Λ࣋ͬͨ "DUJPOT
   PO
   (PPHMFΛ࡞Δ
   !NJ[VUPSZ ਫௗܟຬ
2. ࣗݾ঺հ
 ├── ਫௗܟຬ @mizutory ├── Goldrush Computing୅ද └── ϓϩάϥϚʔ ├── Android ├── iOS ├── Actions On Google ├── Vue.js └── Python
3. "DUJPOͱ͸ (PPHMF
   /FTU
   )VC
   (PPHMF
   )PNF
   ͋Δ͍͸"OESPJEεϚʔτϑΥϯͷ (PPHMF
   "TTJTUBOUͰಈ࡞͢ΔԻ੠Ͱͷର࿩ͰϢʔβʔͱ΍ΓͱΓΛ͢ΔΞϓϦ ͷ͜ͱΛʮΞΫγϣϯʯ͋Δ͍͸ʮ"DUJPOT
   PO
   (PPHMFʯͱݴ͍·͢ɻ
4. "DUJPOಈ࡞·Ͱͷ࢓૊Έ "DUJPO w "TTJTUBOU
   4QFFDI
   UP
   5FYU
   w %JBMPH'MPX
   
   ςΩετʹͳͬͨձ࿩ͷड͚ࡼɺΩʔϫʔυநग़
   w 8FCIPPL
   
   ձ࿩ͷड͚ࡼ͝ͱʹɺฦ౴ͷ࡞੒
5. *OUFOU *OUFOUɿ̍ͭͷձ࿩ͷड͚ࡼ 5SBJOJOH
   QISBTFT
   Ϣʔβʔ͕஻ͬͯ͘ΔͰ͋Ζ͏ձ࿩ͷ૝ఆͰ͢ɻ
6. &OUJUZ &OUJUZ
   
   நग़͍ͨ͠Ωʔϫʔυͷఆٛू
7. *OUFOUͱ&OUJUZͰձ࿩Λड͚Δ
8. 8FCIPPL
   ड͚ͨձ࿩ΛIPPLͯ͠ฦ౴Λੜ੒
9. 8FCIPPL
   
   'VMpMMNFOU
   
   'JSFCBTF
   'VODUJPOT
   
   ˠ
   
   ฦ౴Λੜ੒͢Δ8FC
   "1* *OUFOU
   
   ˠ
   
   ̍ͭ̍ͭͷձ࿩ͷड͚ࡼ &OUJUZ
   
   ˠ
   
   நग़͍ͨ͠Ωʔϫʔυͷఆٛू ʢ8FCIPPLɿ8FCଆͰϦΫΤετΛҾֻ͚ͬΔʣ ʢ'VMMpMMNFOU
   ˠ
   ฦ౴ΛpMMͯ͋͛͠Δ΋ͷ
10. 'JSFCBTF
    'VODUJPOTͷ࡞Γํ $ firebase init Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook
11. parrot ├── firebase.json └── functions ├── index.js ├── node_modules ├── package-lock.json └── package.json *OUFOUϋϯυϥʔ
12. 'use strict'; // Actions on Google client library͔ΒɺDialogflowϞδϡʔϧΛΠϯϙʔτ͢Δ const {dialogflow} = require('actions-on-google'); // firebase-functions packageΛΠϯϙʔτ͢Δ const functions = require('firebase-functions'); // Dialogflow clientΠϯελϯεΛ࡞Δ const app = dialogflow({debug: true}); // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, fish, any}) => { if (animal) { conv.close(animal + '͸ಈ෺Ͱ͢Ͷ'); }else if (fish) { conv.close(fish + '͸ڕͰ͢Ͷ'); }else { conv.close(any + '͸ಈ෺Ͱ΋ڕͰ΋͋Γ·ͤΜ'); } }); // DialogflowApp ΦϒδΣΫτΛHTTPS POSTϦΫΤετͷϋϯυϥʔͱͯ͠ొ࿥ exports.dialogflowFirebaseFulfillment = functions.https.onRequest(app); *OUFOUϋϯυϥʔͷྫ *OUFOU໊ ύϥϝʔλʔ໊ʢ&OUJUZ໊ʣ *OUFOU
    ˠɹձ࿩ͷड͚ࡼɹˠɹϋϯυϥʔ &OUJUZ
    ˠɹநग़͍ͨ͠Ωʔϫʔυɹˠɹύϥϝʔλʔ
13. ಈ෺൛ࢁखઢήʔϜ ಈ෺͠ΓͱΓ *OUFSBDUJWF
    $BOWBT
14. *OUFOUͷ४උ
15. *OUFSBDUJWF
    $BOWBTΛߟྀͨ͠ 'JSFCBTF
    'VODUJPOTͷ࡞Γํ $ firebase init Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ -> Interactive Canvas
16. parrot ├── firebase.json ├── functions │ ├── index.js │ ├── node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html └── index.html functionsϑΥϧμ → Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook publicϑΥϧμ → Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ -> Interactive Canvas
17. // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, pass, any}) => { if (animal) { var result = animals.indexOf(animal); …ɹաڈʹಡΈ্͛ͨಈ෺͡Όͳ͍͔ͷνΣοΫ … //ࠓ౓͸AssistantͷಡΈ্͛Δಈ෺ΛϥϯμϜʹબͿ let newAnimal = getRandomMember(animals); …ɹաڈʹಡΈ্͛ͨಈ෺͡Όͳ͍͔ͷνΣοΫ … //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); }else { conv.close(any + '͸ಈ෺Ͱ͸͋Γ·ͤΜɻ͋ͳͨͷෛ͚Ͱ͢'); consumedAnimals.length = 0; } }); *OUFSBDUJWF
    $BOWBT༻ͷϋϯυϥʔ
18. //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://$ {firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); )UNM3FTQPOTFΫϥεͷΦϒδΣΫτΛฦ͢
    VSM
    
    ϑϩϯτΤϯυ JOEFYIUNM ͕ஔ͍ͯ͋Δ৔ॴ
    EBUB
    
    ϑϩϯτΤϯυʹૹΔσʔλ
20. 8FCIPPL͔ΒͷϨεϙϯεΛݟͯΈΔ
21. { "payload": { "google": { "expectUserResponse": true, "richResponse": { "items": [ { "htmlResponse": { "url": "https://parrot-9855a.firebaseapp.com/", "updatedState": { "scene": "animal", "name": "γϩΫϚ", "picture": "polarbear.jpg" } } }, { "simpleResponse": { "textToSpeech": "γϩΫϚ" } } ] } } } } (PPHMF
    /FTU
    )VCͰ͸͜ΕΛ ड͚औͬͯɺ
    ΢Σϒϖʔδͷϩʔυͱ
    ൃ࿩Λߦ͏ 8FCIPPL͔ΒͷϨεϙϯεΛݟͯΈΔ
22. //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } function hasInteractiveCanvas(conv) { const hasInteractiveCanvas = ɹɹɹɹɹɹɹɹɹ conv.surface.capabilities.has('actions.capability.INTERACTIVE_CANVAS'); return hasInteractiveCanvas } ஫ʣσόΠε͕*OUFSBDUJWF
    $BOWBTͷػೳΛ࣋ͬͯ ͍Δͱ͖͚ͩɺ)UNM3FTQPOTFΛฦ͢Α͏ʹ͢Δ
23. parrot ├── firebase.json ├── functions │ ├── index.js │ ├── node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html └── index.html functionsϑΥϧμ → Functions: node.jsͷweb APIΛ࡞Δػೳɹ→ɹWebhook publicϑΥϧμ → Hosting: html΍cssͳͲͷWebϖʔδͷΞηοτΛϗεςΟϯά͢Δػೳ -> Interactive Canvas ͕͜͜ϩʔυ͞ΕΔ 8FCϖʔδ
24. parrot ├── firebase.json ├── functions │ ├── animal.js │ ├── index.js │ ├── node_modules │ ├── package-lock.json │ └── package.json └── public ├── 404.html ├── css │ └── index.css ├── images │ ├── africaboar.jpg │ ├── africaelephant.jpg │ ├── akagitsune.jpg │ ├── alpaca.jpg │ └── zou.jpg ├── index.html └── js └── index.js 'JSFCBTF
    )PTUJOH޲͚ϑ ϩϯτΤϯυϓϩδΣΫτ ͷߏ଄
25. JOEFYIUNM <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <title>[؆қ൛]ಈ෺͠ΓͱΓ</title> <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;," /> <link rel="stylesheet" href="css/index.css" /> <script src="https://www.gstatic.com/assistant/interactivecanvas/api/ interactive_canvas.min.js"></script> </head> <body> <div class="container"> <div id="animal"> <div id="image_block"> <img id="animalimage" /> </div> <div id="name"> </div> </div> </div> <script src="js/index.js"></script> </body> </html>
26. JOEFYKT window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data) } }) } function onDataReceived(data){ if (data.scene === 'animal') { if(data.picture){ document.querySelector('#animalimage').src = `images/${data.picture}`; }else{ document.querySelector('#animalimage').src = `images/no_image.jpg`; } document.querySelector('#name').innerText = `${data.name}`; } } JOUFSBDUJWF$BOWBTʹSFBEZΠϕϯ τʹϦεφʔΛొ࿥͠ɺPO6QEBUF ίʔϧόοΫΛొ࿥͢Δ
28. σόοά
29. σόοάɿ'JSFCBTF
    )PTUJOH ϩʔΧϧͰͷϨΠΞ΢τͷݕূ
    ˠQVCMJDJOEFYIUNMΛ։͘
    μϛʔσʔλΛೖΕͯͷݕূ
    σΟϓϩΠޙ͸ɺ
    IUUQTϓϩδΣΫτ*%pSFCBTFBQQDPN
    Λ։͍ͯ֬ೝ
    
30. μϛʔσʔλΛ࢖ͬͯͷݕূJOEFYKT window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data) } }) //Debug༻ onDataReceived({ scene: 'animal', picture: "alpaca.jpg", name: "ΞϧύΧ" }) } function onDataReceived(data){ if (data.scene === 'animal') { if(data.picture){ document.querySelector('#animalimage').src = `images/${data.picture}`; }else{ document.querySelector('#animalimage').src = `images/no_image.jpg`; } document.querySelector('#name').innerText = `${data.name}`; } } ϑϩϯτΤϯυίʔυͷσόοά
31. σόοάɿ'JSFCBTF
    'VODUJPOT "DUJPOT
    0O
    (PPHMFγϛϡʔϨʔλʔͷ
    &3303%&#6(3&410/4&λϒΛݟΔ 'JSFCBTF
    $POTPMF
    
    
    'VODUJPOTͷϩάΛݟΔ
33. "DUJPOT
    PO
    (PPHMF
    4JNVMBUPS
    
    &33034
    %&#6(
    3&410/4&
34. 'JSFCBTF
    $POTPMF
    ˠ
    'VODUJPOT
    ˠ
    ϩά
35. ݁߹෦෼ͷςετ
    ˠ"DUJPOT
    PO
    (PPHMF
    4JNVMBUPSΛσΟϕϩούʔπʔϧͰݟΔ
36. 5JQT
38. ݩը૾
39. JOEFYKTɿϔομྖҬͷ֬อ window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data) } }) interactiveCanvas.getHeaderHeightPx().then((height) => { document.body.style.paddingTop = `${height}px`; }) }
41. *OUFSBDUJPO
42. 1BTTϘλϯ
43. JOEFYIUNM <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <title>[؆қ൛]ಈ෺͠ΓͱΓ</title> <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;," /> <link rel="stylesheet" href="css/index.css" /> <script src="https://www.gstatic.com/assistant/interactivecanvas/api/ interactive_canvas.min.js"></script> </head> <body> <div class="container"> <div id="animal"> <div id="image_block"> <img id="animalimage" /> </div> <div id="name"> </div> <button id="pass_button">PASS</button> </div> </div> <script src="js/index.js"></script> </body> </html>
44. JOEFYKTɿϢʔβʔΠϯλϥΫγϣϯ window.onload = () => { interactiveCanvas.ready({ onUpdate(data) { onDataReceived(data) } }) document.querySelector('#pass_button').addEventListener('click', elem => { interactiveCanvas.sendTextQuery("ύε"); }); interactiveCanvas.getHeaderHeightPx().then((height) => { document.body.style.paddingTop = `${height}px`; }) } zύεzΫΤϦʔͷૹ৴
45. 1"44ϘλϯΛԡͨ͜͠ͱ͕ʮύεʯ ͱ࿩͔͚ͨ͜͠ͱͱಉ͡ʹͳΔ
46. // 'what_is_this'ͱ͍͏Intentͷϋϯυϥʔͷ࣮૷ app.intent('what_is_this', (conv, {animal, pass, any}) => { if (animal) { }else if(pass){ //AssistantͷಡΈ্͛Δಈ෺ΛϥϯμϜʹબͿ let newAnimal = getRandomMember(animals); //աڈʹಡΈ্͛ͨΞχϚϧΛௐ΂Δ //Interactive CanvasʹରԠ͍ͯ͠Ε͹HtmlResponseΛฦ͢ if(hasInteractiveCanvas(conv)){ conv.ask(new HtmlResponse({ url: `https://${firebaseConfig.projectId}.firebaseapp.com/`, data: { scene: 'animal', name: newAnimal, picture: animalsPics[newAnimal] } })); } //Assistant͕ಈ෺Λ౴͑Δ conv.ask(newAnimal); }else { conv.close(any + '͸ಈ෺Ͱ͸͋Γ·ͤΜɻ͋ͳͨͷෛ͚Ͱ͢'); consumedAnimals.length = 0; } });
48. OPUFDPNNJ[VUPSZ ຊ೔આ໌ͨ͠ϚςϦΞϧ͸ˣˣˣͰ΋ެ։͍ͯ͠·͢ɻ
49. w "DUJPOT
    PO
    (PPHMF
    w J04ɺ"OESPJEΞϓϦ
    w 8FCɺϑϩϯτΤϯυʢ7VFKTʣ
    w 8FC
    "1*ɺόοΫΤϯυʢOPEFKT
    1ZUIPO%KBOHP
    w ػցֶश
    w *05ɺϩϘοτ "DUJPOʹؔ͢Δ࣭໰΍ˣˣˣͷ։ൃͷ૬ஊ͸ɺɺ NJ[VUPSJ!HPMESVTIDPNQVUJOHDPN
    ͪ͜Β·Ͱˠ ʮ%SPJE,BJHJΛݟͨʯͱݴͬͯ΋Β͑Δͱॿ͔Γ·͢ʂ !NJ[VUPSZ