Creating a Voice- Driven TV Remote with Azure and Alexa Greg Shackles Principal Engineer, Olo @gregshackles.com @gshackles 

Amazon Echo Logitech Harmony Elite & Hub 

Logitech’s Alexa Skill Alexa, turn on the TV Alexa, turn on ESPN Alexa, turn on AppleTV 

We Can Do Better 

Project Goals Change channels by searching listings Finally use that Raspberry Pi I bought 5 years ago No other servers/infrastructure to manage Use F# and Azure As cheap as possible 

Getting the Data 

XML TV Listings $20 / year xmltvlistings.com 

Azure SQL Database Smallest instance $5 / month 

Channel Format CBS (WCBS) New York, NY WCBS-TV1 2 newyork.cbslocal.com/ 

Parsing the Data type Channels = XmlProvider<"./channels.xml"> let doc = Channels.Parse (File.ReadAllText "./channels.xml") type TvChannel = { XmlTvId: string; FullName: string; DisplayName: string; Number: int } 

Loading Into SQL doc.Channels |> Array.map (fun channel -> match channel.DisplayNames with | [| fullName; displayName; number |] -> Some({ XmlTvId = channel.Id FullName = fullName.String.Value DisplayName = Regex.Replace(displayName.String.Value, "\\-HD$", "", RegexOptions.None) Number = number.Number.Value }) | _ -> None ) |> Array.choose id |> Array.filter (fun channel -> channel.Number > 500 && channel.Number < 1000) |> Array.iter (fun channel -> use cmd = new SqlCommandProvider< "INSERT INTO Channel VALUES (@xmlTvId, @displayName, @fullName, @number)", "name=TVListings">() cmd.Execute(xmlTvId = channel.XmlTvId, displayName = channel.DisplayName, fullName = channel.FullName, number = channel.Number) |> ignore printfn "Added channel: %s" channel.DisplayName ) 

DownloadLineup open FSharp.Data let Run(timerTrigger: TimerInfo, log: TraceWriter, lineupsBlob: byref) = let lineupUrl = sprintf "https://www.xmltvlistings.com/xmltv/get/%s/%s/2/0" (Environment.GetEnvironmentVariable("XmlTvApiKey")) (Environment.GetEnvironmentVariable("XmlTvLineupId")) let xml = Http.RequestString(lineupUrl, responseEncodingOverride = "utf-8") lineupsBlob <- xml sprintf "Successfully downloaded XML, length: %i" xml.Length |> log.Info Runs every morning 

ImportLineup type Listings = XmlProvider<"D:\\home\\site\\wwwroot\\ImportLineup\\schedule-sample.xml"> let addShow (log: TraceWriter) (channelLookup: IDictionary) (show: Listings.Programme) = match channelLookup.TryGetValue show.Channel with | (true, channelId) -> let title = match (show.Title.Value, show.SubTitle) with | "Movie", Some(subtitle) -> subtitle.Value.Trim() | title, _ -> title.Trim() let description = if show.Desc.IsSome then show.Desc.Value.Value else "" Database.addShow title show.Start show.Stop channelId description show.Category.Value sprintf "Added show : %s" title |> log.Info | (false, _) -> () Runs when a new file arrives in blob storage 

Adding Search 

Azure Search Service Created indexes for channels shows Both sourced from SQL Server Free tier: 50MB, 10K documents 

Searching for a Show 

Search Service Usage 

Re-Indexing 

Modifying ImportLineup { "type": "queue", "name": "outputQueueItem", "queueName": "rebuild-requests", "connection": "tvlistingstorage_STORAGE", "direction": "out" } let Run(xml: string, name: string, log: TraceWriter, outputQueueItem: byref) = // ... outputQueueItem <- name Write to queue when it finishes importing: 

RebuildSearchIndex type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top": 100, "search": "seinf type SearchResults = JsonProvider<""" {"value": [{ "ShowId": "abc" }]} """> type DeleteRequest = JsonProvider<""" {"value": [{ "@search.action": "delete", "ShowId": "abc" }] let search query filter count = SearchRequest.Root(filter, count, query).JsonValue.ToString() |> postJson "indexes('shows')/docs/search" |> SearchResults.Parse |> fun results -> results.Value let rec clearIndex() = let results = search "" "" 1000 if not <| Seq.isEmpty results then results |> Array.map (fun result -> DeleteRequest.Value("delete", result.ShowId)) |> fun ops -> DeleteRequest.Root(ops).JsonValue.ToString() |> postJson "indexes('shows')/docs/index" |> ignore clearIndex() let rebuildIndex() = postJson "indexers/shows-indexer/run" "" |> ignore 

The Device API 

Harmony API Node server Query/control Harmony Hubs Protocols HTTP MQTT github.com/maddox/harmony-api 

Exposing the API Registered shackles.house Certificate through Let's Encrypt No-IP $35 / year for custom domain support Agent running on the Raspberry Pi Proxy through NGINX 

Let's Encrypt certbot-auto certonly -a webroot --webroot-path=/var/www/html -d shackles.house -d www.shackles.house 0 3 * * 1 /usr/bin/certbot-auto renew >> /var/log/certbot-renew.log 5 3 * * 1 /etc/init.d/nginx restart Renew the certificate every week: 

NGINX Configuration limit_req_zone $binary_remote_addr zone=harmonyzone:10m rate=1r/s; server { listen 9000 ssl default_server; listen [::]:9000 ssl default_server; # security configuration omitted, see blog for details if ($http_authorization != 'letmein') { return 401; } location / { limit_req zone=harmonyzone burst=5; proxy_pass http://127.0.0.1:8282; proxy_redirect off; } } 

Alexa, tell the TV to mute 

DirectCommand Intent { "intents": [ { "intent": "DirectCommand", "slots": [ { "name": "command", "type": "command_name" }, { "name": "target", "type": "command_target" } ] } ] } DirectCommand {command} DirectCommand {command} the {target} DirectCommand {command} {target} Sample utterances: 

RemoteSkill let executeCommand commandSlug = let url = sprintf "%s/commands/%s" baseUrl commandSlug let authHeader = "Authorization", apiKey Http.RequestString(url, headers = [authHeader], httpMethod = "POST") |> ignore let handleDirectCommand (intent: Intent) = match (Commands.getCommand intent.Slots.["command"].Value) with | Some(command) -> Commands.executeCommand command.Slug buildResponse "OK" true | None -> buildResponse "Sorry, that command is not available right now" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true 

RemoteSkill type RemoteSpeechlet(log: TraceWriter) = inherit Speechlet() override this.OnIntent(request: IntentRequest, session: Session): SpeechletResponse = sprintf "OnIntent: request %s, session %s" request.RequestId session.SessionId |> log.Info handleIntent request.Intent let Run(req: HttpRequestMessage, log: TraceWriter) = let speechlet = RemoteSpeechlet log speechlet.GetResponse req AlexaSkillsKit: github.com/AreYouFreeBusy/AlexaSkillsKit.NET 

Alexa, tell the TV to turn on the Yankees game 

Changing the Channel let changeChannel number = string number |> Seq.map string |> Seq.iter executeCommand executeCommand "select" 

Querying Search Service type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top": 100, "search": "sei type ShowSearchResults = JsonProvider<""" {"value": [{"@search.score": 2.5086286, "ShowId": "10 "StartTime": "20170102200000", "EndTime": "Description": "Yada yada yada", "Categor type ChannelSearchResults = JsonProvider<""" {"value": [{"@search.score": 1, "ChannelId": "150" "DisplayName": "CBSSN-HD", "FullName": let private search index (request: SearchRequest.Root) = let url = sprintf "%s/indexes('%s')/docs/search?api-version=2015-02-28" baseUrl index let apiKeyHeader = "api-key", apiKey let json = request.JsonValue.ToString() Http.RequestString(url, body = TextRequest json, headers = [ ContentType HttpContentTypes.Json; apiKeyHeader ]) let private searchShows request = search "shows" request |> ShowSearchResults.Parse let private searchChannels request = search "channels" request |> ChannelSearchResults.Parse 

Finding a Show let findShowOnNow (name: string) = let timestamp = DateTime.UtcNow.ToString("yyyyMMddHHmmss") let filter = sprintf "StartTime lt '%s' and EndTime gt '%s'" timestamp timestamp SearchRequest.Root(filter, 1, name) |> searchShows |> fun result -> Array.tryPick Some result.Value 

WatchShow Intent { "intent": "WatchShow", "slots": [ { "name": "name", "type": "show_name" } ] } WatchShow turn on {name} WatchShow watch {name} WatchShow switch to {name} Sample utterances: 

RemoteSkill let handleWatchShow (intent: Intent) = Search.findShowOnNow intent.Slots.["name"].Value |> function | Some(show) -> Search.findChannel show.ChannelId |> function | Some(channel) -> Commands.changeChannel channel buildResponse "OK" true | None -> buildResponse "Sorry, I could not find the channel for that show" true | None -> buildResponse "Sorry, I could not find that show" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | "WatchShow" -> handleWatchShow intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true 

Switching from HTTP to MQTT 

The HTTP API Had Some Downsides Exposed a public API to my living room Performance kind of sucked A lot of moving parts No-IP NGINX Let's Encrypt 

MQTT Message Queue Telemetry Transport Lightweight publish/subscribe protocol Built on TCP protocol Commonly used in IoT scenarios Supported by Azure IoT Hub 

IoT Hub Bridge const mqttServer = require('net').createServer(broker.handle); mqttServer.listen(1883, () => console.log('MQTT broker listening')); function onAzureConnect(err) { azureClient.on('error', console.error); azureClient.on('disconnect', function () { azureClient.removeAllListeners(); azureClient.open(onAzureConnect); }); function sendCommand(command) { const [topic, payload] = command.split(';'); broker.publish({ topic, payload }); } azureClient.on('message', msg => msg.data.toString() .split('\n') .forEach((command, index) => setTimeout( () => sendCommand(command), index * CommandDelayMs))); } azureClient.open(onAzureConnect); 

Sending a Command 

RemoteSkill let executeCommands commandSlugs = async { commandSlugs |> Seq.map (fun slug -> sprintf "harmony-api/hubs/living-room/command;%s" slug) |> String.concat "\n" |> Encoding.ASCII.GetBytes |> fun bytes -> new Message(bytes) |> fun message -> serviceClient.SendAsync("harmony-bridge", message) |> Async.AwaitIAsyncResult |> Async.Ignore |> ignore } |> Async.RunSynchronously let executeCommand commandSlug = executeCommands [commandSlug] 

Persisting Available Commands CREATE TABLE AvailableCommand( AvailableCommandId int IDENTITY(1,1) NOT NULL, Name nvarchar(32) NOT NULL, Slug nvarchar(16) NOT NULL, Label nvarchar(32) NOT NULL ) broker.on('publish', packet => { if (packet.topic !== 'harmony-api/hubs/living-room/current_activity') { return; } request({ url: 'http://localhost:8282/hubs/living-room/commands', json: true }) .then(res => updateAvailableCommands(res.commands)); }); 

Monitoring Performance 

Application Insights 

Timing Sub-Operations let private telemetryClient = TelemetryClient(InstrumentationKey = instrumentationKey) let setOperationId operationId = telemetryClient.Context.Operation.Id <- operationId let startOperation (name:string) = telemetryClient.StartOperation(name) let private searchShows request = use operation = Telemetry.startOperation "ShowSearch" search "shows" request |> ShowSearchResults.Parse let private searchChannels request = use operation = Telemetry.startOperation "ChannelSearch" search "channels" request |> ChannelSearchResults.Parse let executeCommand commandSlug = use operation = Telemetry.startOperation "ExecuteCommand" // ... 

Viewing Performance 

What's The Cost? 

Total: $7.56 Costs Component Monthly Cost XML TV Listings $1.67 SQL Server $4.99 Storage $0.63 App Service $0.27 IoT Hub Free Search Service Free Application Insights Free 

Let's Check It Out 

Questions? Greg Shackles Principal Engineer, Olo gregshackles.com @gshackles gregshackles.com/tag/remote