Creating a Voice-Driven TV Remote with Azure and Alexa

279b474d14f72e4daa1fc76e6f3c929f?s=47 Greg Shackles
September 25, 2017

Creating a Voice-Driven TV Remote with Azure and Alexa

279b474d14f72e4daa1fc76e6f3c929f?s=128

Greg Shackles

September 25, 2017
Tweet

Transcript

  1. Creating a Voice- Driven TV Remote with Azure and Alexa

    Greg Shackles Principal Engineer, Olo @gregshackles.com @gshackles
  2. Amazon Echo Logitech Harmony Elite & Hub

  3. Logitech’s Alexa Skill Alexa, turn on the TV Alexa, turn

    on ESPN Alexa, turn on AppleTV
  4. We Can Do Better

  5. Project Goals Change channels by searching listings Finally use that

    Raspberry Pi I bought 5 years ago No other servers/infrastructure to manage Use F# and Azure As cheap as possible
  6. Getting the Data

  7. XML TV Listings $20 / year xmltvlistings.com

  8. Azure SQL Database Smallest instance $5 / month

  9. Channel Format <?xml version=“1.0” encoding=“UTF-8” ?> <tv date=“12/03/2016” source-info-url=“http://www.tvmedia.ca” source-info-name=“TV

    Media”> <channel id=“1766.stations.xmltv.tvmedia.ca”> <display-name>CBS (WCBS) New York, NY</display-name> <display-name>WCBS-TV1</display-name> <display-name>2</display-name> <url>newyork.cbslocal.com/</url> <icon src=“http://cdn.tvpassport.com/image/station/76x28/cbs.png”/> </channel> </tv>
  10. Parsing the Data type Channels = XmlProvider<"./channels.xml"> let doc =

    Channels.Parse (File.ReadAllText "./channels.xml") type TvChannel = { XmlTvId: string; FullName: string; DisplayName: string; Number: int }
  11. Loading Into SQL doc.Channels |> Array.map (fun channel -> match

    channel.DisplayNames with | [| fullName; displayName; number |] -> Some({ XmlTvId = channel.Id FullName = fullName.String.Value DisplayName = Regex.Replace(displayName.String.Value, "\\-HD$", "", RegexOptions.None) Number = number.Number.Value }) | _ -> None ) |> Array.choose id |> Array.filter (fun channel -> channel.Number > 500 && channel.Number < 1000) |> Array.iter (fun channel -> use cmd = new SqlCommandProvider< "INSERT INTO Channel VALUES (@xmlTvId, @displayName, @fullName, @number)", "name=TVListings">() cmd.Execute(xmlTvId = channel.XmlTvId, displayName = channel.DisplayName, fullName = channel.FullName, number = channel.Number) |> ignore printfn "Added channel: %s" channel.DisplayName )
  12. DownloadLineup open FSharp.Data let Run(timerTrigger: TimerInfo, log: TraceWriter, lineupsBlob: byref<string>)

    = let lineupUrl = sprintf "https://www.xmltvlistings.com/xmltv/get/%s/%s/2/0" (Environment.GetEnvironmentVariable("XmlTvApiKey")) (Environment.GetEnvironmentVariable("XmlTvLineupId")) let xml = Http.RequestString(lineupUrl, responseEncodingOverride = "utf-8") lineupsBlob <- xml sprintf "Successfully downloaded XML, length: %i" xml.Length |> log.Info Runs every morning
  13. ImportLineup type Listings = XmlProvider<"D:\\home\\site\\wwwroot\\ImportLineup\\schedule-sample.xml"> let addShow (log: TraceWriter) (channelLookup:

    IDictionary<string, int>) (show: Listings.Programme) = match channelLookup.TryGetValue show.Channel with | (true, channelId) -> let title = match (show.Title.Value, show.SubTitle) with | "Movie", Some(subtitle) -> subtitle.Value.Trim() | title, _ -> title.Trim() let description = if show.Desc.IsSome then show.Desc.Value.Value else "" Database.addShow title show.Start show.Stop channelId description show.Category.Value sprintf "Added show : %s" title |> log.Info | (false, _) -> () Runs when a new file arrives in blob storage
  14. Adding Search

  15. Azure Search Service Created indexes for channels shows Both sourced

    from SQL Server Free tier: 50MB, 10K documents
  16. Searching for a Show

  17. Search Service Usage

  18. Re-Indexing

  19. Modifying ImportLineup { "type": "queue", "name": "outputQueueItem", "queueName": "rebuild-requests", "connection":

    "tvlistingstorage_STORAGE", "direction": "out" } let Run(xml: string, name: string, log: TraceWriter, outputQueueItem: byref<string>) = // ... outputQueueItem <- name Write to queue when it finishes importing:
  20. RebuildSearchIndex type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top":

    100, "search": "seinf type SearchResults = JsonProvider<""" {"value": [{ "ShowId": "abc" }]} """> type DeleteRequest = JsonProvider<""" {"value": [{ "@search.action": "delete", "ShowId": "abc" }] let search query filter count = SearchRequest.Root(filter, count, query).JsonValue.ToString() |> postJson "indexes('shows')/docs/search" |> SearchResults.Parse |> fun results -> results.Value let rec clearIndex() = let results = search "" "" 1000 if not <| Seq.isEmpty results then results |> Array.map (fun result -> DeleteRequest.Value("delete", result.ShowId)) |> fun ops -> DeleteRequest.Root(ops).JsonValue.ToString() |> postJson "indexes('shows')/docs/index" |> ignore clearIndex() let rebuildIndex() = postJson "indexers/shows-indexer/run" "" |> ignore
  21. The Device API

  22. Harmony API Node server Query/control Harmony Hubs Protocols HTTP MQTT

    github.com/maddox/harmony-api
  23. Exposing the API Registered shackles.house Certificate through Let's Encrypt No-IP

    $35 / year for custom domain support Agent running on the Raspberry Pi Proxy through NGINX
  24. Let's Encrypt certbot-auto certonly -a webroot --webroot-path=/var/www/html -d shackles.house -d

    www.shackles.house 0 3 * * 1 /usr/bin/certbot-auto renew >> /var/log/certbot-renew.log 5 3 * * 1 /etc/init.d/nginx restart Renew the certificate every week:
  25. NGINX Configuration limit_req_zone $binary_remote_addr zone=harmonyzone:10m rate=1r/s; server { listen 9000

    ssl default_server; listen [::]:9000 ssl default_server; # security configuration omitted, see blog for details if ($http_authorization != 'letmein') { return 401; } location / { limit_req zone=harmonyzone burst=5; proxy_pass http://127.0.0.1:8282; proxy_redirect off; } }
  26. Alexa, tell the TV to mute

  27. DirectCommand Intent { "intents": [ { "intent": "DirectCommand", "slots": [

    { "name": "command", "type": "command_name" }, { "name": "target", "type": "command_target" } ] } ] } DirectCommand {command} DirectCommand {command} the {target} DirectCommand {command} {target} Sample utterances:
  28. RemoteSkill let executeCommand commandSlug = let url = sprintf "%s/commands/%s"

    baseUrl commandSlug let authHeader = "Authorization", apiKey Http.RequestString(url, headers = [authHeader], httpMethod = "POST") |> ignore let handleDirectCommand (intent: Intent) = match (Commands.getCommand intent.Slots.["command"].Value) with | Some(command) -> Commands.executeCommand command.Slug buildResponse "OK" true | None -> buildResponse "Sorry, that command is not available right now" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
  29. RemoteSkill type RemoteSpeechlet(log: TraceWriter) = inherit Speechlet() override this.OnIntent(request: IntentRequest,

    session: Session): SpeechletResponse = sprintf "OnIntent: request %s, session %s" request.RequestId session.SessionId |> log.Info handleIntent request.Intent let Run(req: HttpRequestMessage, log: TraceWriter) = let speechlet = RemoteSpeechlet log speechlet.GetResponse req AlexaSkillsKit: github.com/AreYouFreeBusy/AlexaSkillsKit.NET
  30. Alexa, tell the TV to turn on the Yankees game

  31. Changing the Channel let changeChannel number = string number |>

    Seq.map string |> Seq.iter executeCommand executeCommand "select"
  32. Querying Search Service type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq

    125", "top": 100, "search": "sei type ShowSearchResults = JsonProvider<""" {"value": [{"@search.score": 2.5086286, "ShowId": "10 "StartTime": "20170102200000", "EndTime": "Description": "Yada yada yada", "Categor type ChannelSearchResults = JsonProvider<""" {"value": [{"@search.score": 1, "ChannelId": "150" "DisplayName": "CBSSN-HD", "FullName": let private search index (request: SearchRequest.Root) = let url = sprintf "%s/indexes('%s')/docs/search?api-version=2015-02-28" baseUrl index let apiKeyHeader = "api-key", apiKey let json = request.JsonValue.ToString() Http.RequestString(url, body = TextRequest json, headers = [ ContentType HttpContentTypes.Json; apiKeyHeader ]) let private searchShows request = search "shows" request |> ShowSearchResults.Parse let private searchChannels request = search "channels" request |> ChannelSearchResults.Parse
  33. Finding a Show let findShowOnNow (name: string) = let timestamp

    = DateTime.UtcNow.ToString("yyyyMMddHHmmss") let filter = sprintf "StartTime lt '%s' and EndTime gt '%s'" timestamp timestamp SearchRequest.Root(filter, 1, name) |> searchShows |> fun result -> Array.tryPick Some result.Value
  34. WatchShow Intent { "intent": "WatchShow", "slots": [ { "name": "name",

    "type": "show_name" } ] } WatchShow turn on {name} WatchShow watch {name} WatchShow switch to {name} Sample utterances:
  35. RemoteSkill let handleWatchShow (intent: Intent) = Search.findShowOnNow intent.Slots.["name"].Value |> function

    | Some(show) -> Search.findChannel show.ChannelId |> function | Some(channel) -> Commands.changeChannel channel buildResponse "OK" true | None -> buildResponse "Sorry, I could not find the channel for that show" true | None -> buildResponse "Sorry, I could not find that show" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | "WatchShow" -> handleWatchShow intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
  36. Switching from HTTP to MQTT

  37. The HTTP API Had Some Downsides Exposed a public API

    to my living room Performance kind of sucked A lot of moving parts No-IP NGINX Let's Encrypt
  38. MQTT Message Queue Telemetry Transport Lightweight publish/subscribe protocol Built on

    TCP protocol Commonly used in IoT scenarios Supported by Azure IoT Hub
  39. IoT Hub Bridge const mqttServer = require('net').createServer(broker.handle); mqttServer.listen(1883, () =>

    console.log('MQTT broker listening')); function onAzureConnect(err) { azureClient.on('error', console.error); azureClient.on('disconnect', function () { azureClient.removeAllListeners(); azureClient.open(onAzureConnect); }); function sendCommand(command) { const [topic, payload] = command.split(';'); broker.publish({ topic, payload }); } azureClient.on('message', msg => msg.data.toString() .split('\n') .forEach((command, index) => setTimeout( () => sendCommand(command), index * CommandDelayMs))); } azureClient.open(onAzureConnect);
  40. Sending a Command

  41. RemoteSkill let executeCommands commandSlugs = async { commandSlugs |> Seq.map

    (fun slug -> sprintf "harmony-api/hubs/living-room/command;%s" slug) |> String.concat "\n" |> Encoding.ASCII.GetBytes |> fun bytes -> new Message(bytes) |> fun message -> serviceClient.SendAsync("harmony-bridge", message) |> Async.AwaitIAsyncResult |> Async.Ignore |> ignore } |> Async.RunSynchronously let executeCommand commandSlug = executeCommands [commandSlug]
  42. Persisting Available Commands CREATE TABLE AvailableCommand( AvailableCommandId int IDENTITY(1,1) NOT

    NULL, Name nvarchar(32) NOT NULL, Slug nvarchar(16) NOT NULL, Label nvarchar(32) NOT NULL ) broker.on('publish', packet => { if (packet.topic !== 'harmony-api/hubs/living-room/current_activity') { return; } request({ url: 'http://localhost:8282/hubs/living-room/commands', json: true }) .then(res => updateAvailableCommands(res.commands)); });
  43. Monitoring Performance

  44. Application Insights

  45. Timing Sub-Operations let private telemetryClient = TelemetryClient(InstrumentationKey = instrumentationKey) let

    setOperationId operationId = telemetryClient.Context.Operation.Id <- operationId let startOperation (name:string) = telemetryClient.StartOperation<DependencyTelemetry>(name) let private searchShows request = use operation = Telemetry.startOperation "ShowSearch" search "shows" request |> ShowSearchResults.Parse let private searchChannels request = use operation = Telemetry.startOperation "ChannelSearch" search "channels" request |> ChannelSearchResults.Parse let executeCommand commandSlug = use operation = Telemetry.startOperation "ExecuteCommand" // ...
  46. Viewing Performance

  47. What's The Cost?

  48. Total: $7.56 Costs Component Monthly Cost XML TV Listings $1.67

    SQL Server $4.99 Storage $0.63 App Service $0.27 IoT Hub Free Search Service Free Application Insights Free
  49. Let's Check It Out

  50. Questions? Greg Shackles Principal Engineer, Olo gregshackles.com @gshackles gregshackles.com/tag/remote