Creating a Voice-Driven TV Remote with Azure and Alexa

Transcript

1. Creating a Voice- Driven TV Remote with Azure and Alexa Greg Shackles Principal Engineer, Olo @gregshackles.com @gshackles
2. Amazon Echo Logitech Harmony Elite & Hub
3. Logitech’s Alexa Skill Alexa, turn on the TV Alexa, turn on ESPN Alexa, turn on AppleTV
4. We Can Do Better
5. Project Goals Change channels by searching listings Finally use that Raspberry Pi I bought 5 years ago No other servers/infrastructure to manage Use F# and Azure As cheap as possible
6. Getting the Data
7. XML TV Listings $20 / year xmltvlistings.com
8. Azure SQL Database Smallest instance $5 / month
9. Channel Format <?xml version=“1.0” encoding=“UTF-8” ?> <tv date=“12/03/2016” source-info-url=“http://www.tvmedia.ca” source-info-name=“TV Media”> <channel id=“1766.stations.xmltv.tvmedia.ca”> <display-name>CBS (WCBS) New York, NY</display-name> <display-name>WCBS-TV1</display-name> <display-name>2</display-name> <url>newyork.cbslocal.com/</url> <icon src=“http://cdn.tvpassport.com/image/station/76x28/cbs.png”/> </channel> </tv>
10. Parsing the Data type Channels = XmlProvider<"./channels.xml"> let doc = Channels.Parse (File.ReadAllText "./channels.xml") type TvChannel = { XmlTvId: string; FullName: string; DisplayName: string; Number: int }
11. Loading Into SQL doc.Channels |> Array.map (fun channel -> match channel.DisplayNames with | [| fullName; displayName; number |] -> Some({ XmlTvId = channel.Id FullName = fullName.String.Value DisplayName = Regex.Replace(displayName.String.Value, "\\-HD$", "", RegexOptions.None) Number = number.Number.Value }) | _ -> None ) |> Array.choose id |> Array.filter (fun channel -> channel.Number > 500 && channel.Number < 1000) |> Array.iter (fun channel -> use cmd = new SqlCommandProvider< "INSERT INTO Channel VALUES (@xmlTvId, @displayName, @fullName, @number)", "name=TVListings">() cmd.Execute(xmlTvId = channel.XmlTvId, displayName = channel.DisplayName, fullName = channel.FullName, number = channel.Number) |> ignore printfn "Added channel: %s" channel.DisplayName )
12. DownloadLineup open FSharp.Data let Run(timerTrigger: TimerInfo, log: TraceWriter, lineupsBlob: byref<string>) = let lineupUrl = sprintf "https://www.xmltvlistings.com/xmltv/get/%s/%s/2/0" (Environment.GetEnvironmentVariable("XmlTvApiKey")) (Environment.GetEnvironmentVariable("XmlTvLineupId")) let xml = Http.RequestString(lineupUrl, responseEncodingOverride = "utf-8") lineupsBlob <- xml sprintf "Successfully downloaded XML, length: %i" xml.Length |> log.Info Runs every morning
13. ImportLineup type Listings = XmlProvider<"D:\\home\\site\\wwwroot\\ImportLineup\\schedule-sample.xml"> let addShow (log: TraceWriter) (channelLookup: IDictionary<string, int>) (show: Listings.Programme) = match channelLookup.TryGetValue show.Channel with | (true, channelId) -> let title = match (show.Title.Value, show.SubTitle) with | "Movie", Some(subtitle) -> subtitle.Value.Trim() | title, _ -> title.Trim() let description = if show.Desc.IsSome then show.Desc.Value.Value else "" Database.addShow title show.Start show.Stop channelId description show.Category.Value sprintf "Added show : %s" title |> log.Info | (false, _) -> () Runs when a new file arrives in blob storage
14. Adding Search
15. Azure Search Service Created indexes for channels shows Both sourced from SQL Server Free tier: 50MB, 10K documents
16. Searching for a Show
17. Search Service Usage
18. Re-Indexing
19. Modifying ImportLineup { "type": "queue", "name": "outputQueueItem", "queueName": "rebuild-requests", "connection": "tvlistingstorage_STORAGE", "direction": "out" } let Run(xml: string, name: string, log: TraceWriter, outputQueueItem: byref<string>) = // ... outputQueueItem <- name Write to queue when it finishes importing:
20. RebuildSearchIndex type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top": 100, "search": "seinf type SearchResults = JsonProvider<""" {"value": [{ "ShowId": "abc" }]} """> type DeleteRequest = JsonProvider<""" {"value": [{ "@search.action": "delete", "ShowId": "abc" }] let search query filter count = SearchRequest.Root(filter, count, query).JsonValue.ToString() |> postJson "indexes('shows')/docs/search" |> SearchResults.Parse |> fun results -> results.Value let rec clearIndex() = let results = search "" "" 1000 if not <| Seq.isEmpty results then results |> Array.map (fun result -> DeleteRequest.Value("delete", result.ShowId)) |> fun ops -> DeleteRequest.Root(ops).JsonValue.ToString() |> postJson "indexes('shows')/docs/index" |> ignore clearIndex() let rebuildIndex() = postJson "indexers/shows-indexer/run" "" |> ignore
21. The Device API
22. Harmony API Node server Query/control Harmony Hubs Protocols HTTP MQTT github.com/maddox/harmony-api
23. Exposing the API Registered shackles.house Certificate through Let's Encrypt No-IP $35 / year for custom domain support Agent running on the Raspberry Pi Proxy through NGINX
24. Let's Encrypt certbot-auto certonly -a webroot --webroot-path=/var/www/html -d shackles.house -d www.shackles.house 0 3 * * 1 /usr/bin/certbot-auto renew >> /var/log/certbot-renew.log 5 3 * * 1 /etc/init.d/nginx restart Renew the certificate every week:
25. NGINX Configuration limit_req_zone $binary_remote_addr zone=harmonyzone:10m rate=1r/s; server { listen 9000 ssl default_server; listen [::]:9000 ssl default_server; # security configuration omitted, see blog for details if ($http_authorization != 'letmein') { return 401; } location / { limit_req zone=harmonyzone burst=5; proxy_pass http://127.0.0.1:8282; proxy_redirect off; } }
26. Alexa, tell the TV to mute
27. DirectCommand Intent { "intents": [ { "intent": "DirectCommand", "slots": [ { "name": "command", "type": "command_name" }, { "name": "target", "type": "command_target" } ] } ] } DirectCommand {command} DirectCommand {command} the {target} DirectCommand {command} {target} Sample utterances:
28. RemoteSkill let executeCommand commandSlug = let url = sprintf "%s/commands/%s" baseUrl commandSlug let authHeader = "Authorization", apiKey Http.RequestString(url, headers = [authHeader], httpMethod = "POST") |> ignore let handleDirectCommand (intent: Intent) = match (Commands.getCommand intent.Slots.["command"].Value) with | Some(command) -> Commands.executeCommand command.Slug buildResponse "OK" true | None -> buildResponse "Sorry, that command is not available right now" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
29. RemoteSkill type RemoteSpeechlet(log: TraceWriter) = inherit Speechlet() override this.OnIntent(request: IntentRequest, session: Session): SpeechletResponse = sprintf "OnIntent: request %s, session %s" request.RequestId session.SessionId |> log.Info handleIntent request.Intent let Run(req: HttpRequestMessage, log: TraceWriter) = let speechlet = RemoteSpeechlet log speechlet.GetResponse req AlexaSkillsKit: github.com/AreYouFreeBusy/AlexaSkillsKit.NET
30. Alexa, tell the TV to turn on the Yankees game
31. Changing the Channel let changeChannel number = string number |> Seq.map string |> Seq.iter executeCommand executeCommand "select"
32. Querying Search Service type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top": 100, "search": "sei type ShowSearchResults = JsonProvider<""" {"value": [{"@search.score": 2.5086286, "ShowId": "10 "StartTime": "20170102200000", "EndTime": "Description": "Yada yada yada", "Categor type ChannelSearchResults = JsonProvider<""" {"value": [{"@search.score": 1, "ChannelId": "150" "DisplayName": "CBSSN-HD", "FullName": let private search index (request: SearchRequest.Root) = let url = sprintf "%s/indexes('%s')/docs/search?api-version=2015-02-28" baseUrl index let apiKeyHeader = "api-key", apiKey let json = request.JsonValue.ToString() Http.RequestString(url, body = TextRequest json, headers = [ ContentType HttpContentTypes.Json; apiKeyHeader ]) let private searchShows request = search "shows" request |> ShowSearchResults.Parse let private searchChannels request = search "channels" request |> ChannelSearchResults.Parse
33. Finding a Show let findShowOnNow (name: string) = let timestamp = DateTime.UtcNow.ToString("yyyyMMddHHmmss") let filter = sprintf "StartTime lt '%s' and EndTime gt '%s'" timestamp timestamp SearchRequest.Root(filter, 1, name) |> searchShows |> fun result -> Array.tryPick Some result.Value
34. WatchShow Intent { "intent": "WatchShow", "slots": [ { "name": "name", "type": "show_name" } ] } WatchShow turn on {name} WatchShow watch {name} WatchShow switch to {name} Sample utterances:
35. RemoteSkill let handleWatchShow (intent: Intent) = Search.findShowOnNow intent.Slots.["name"].Value |> function | Some(show) -> Search.findChannel show.ChannelId |> function | Some(channel) -> Commands.changeChannel channel buildResponse "OK" true | None -> buildResponse "Sorry, I could not find the channel for that show" true | None -> buildResponse "Sorry, I could not find that show" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | "WatchShow" -> handleWatchShow intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
36. Switching from HTTP to MQTT
37. The HTTP API Had Some Downsides Exposed a public API to my living room Performance kind of sucked A lot of moving parts No-IP NGINX Let's Encrypt
38. MQTT Message Queue Telemetry Transport Lightweight publish/subscribe protocol Built on TCP protocol Commonly used in IoT scenarios Supported by Azure IoT Hub
39. IoT Hub Bridge const mqttServer = require('net').createServer(broker.handle); mqttServer.listen(1883, () => console.log('MQTT broker listening')); function onAzureConnect(err) { azureClient.on('error', console.error); azureClient.on('disconnect', function () { azureClient.removeAllListeners(); azureClient.open(onAzureConnect); }); function sendCommand(command) { const [topic, payload] = command.split(';'); broker.publish({ topic, payload }); } azureClient.on('message', msg => msg.data.toString() .split('\n') .forEach((command, index) => setTimeout( () => sendCommand(command), index * CommandDelayMs))); } azureClient.open(onAzureConnect);
40. Sending a Command
41. RemoteSkill let executeCommands commandSlugs = async { commandSlugs |> Seq.map (fun slug -> sprintf "harmony-api/hubs/living-room/command;%s" slug) |> String.concat "\n" |> Encoding.ASCII.GetBytes |> fun bytes -> new Message(bytes) |> fun message -> serviceClient.SendAsync("harmony-bridge", message) |> Async.AwaitIAsyncResult |> Async.Ignore |> ignore } |> Async.RunSynchronously let executeCommand commandSlug = executeCommands [commandSlug]
42. Persisting Available Commands CREATE TABLE AvailableCommand( AvailableCommandId int IDENTITY(1,1) NOT NULL, Name nvarchar(32) NOT NULL, Slug nvarchar(16) NOT NULL, Label nvarchar(32) NOT NULL ) broker.on('publish', packet => { if (packet.topic !== 'harmony-api/hubs/living-room/current_activity') { return; } request({ url: 'http://localhost:8282/hubs/living-room/commands', json: true }) .then(res => updateAvailableCommands(res.commands)); });
43. Monitoring Performance
44. Application Insights
45. Timing Sub-Operations let private telemetryClient = TelemetryClient(InstrumentationKey = instrumentationKey) let setOperationId operationId = telemetryClient.Context.Operation.Id <- operationId let startOperation (name:string) = telemetryClient.StartOperation<DependencyTelemetry>(name) let private searchShows request = use operation = Telemetry.startOperation "ShowSearch" search "shows" request |> ShowSearchResults.Parse let private searchChannels request = use operation = Telemetry.startOperation "ChannelSearch" search "channels" request |> ChannelSearchResults.Parse let executeCommand commandSlug = use operation = Telemetry.startOperation "ExecuteCommand" // ...
46. Viewing Performance
47. What's The Cost?
48. Total: $7.56 Costs Component Monthly Cost XML TV Listings $1.67 SQL Server $4.99 Storage $0.63 App Service $0.27 IoT Hub Free Search Service Free Application Insights Free
49. Let's Check It Out
50. Questions? Greg Shackles Principal Engineer, Olo gregshackles.com @gshackles gregshackles.com/tag/remote