Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Creating a Voice-Driven TV Remote with Azure and Alexa

Greg Shackles
September 25, 2017

Creating a Voice-Driven TV Remote with Azure and Alexa

Greg Shackles

September 25, 2017
Tweet

More Decks by Greg Shackles

Other Decks in Technology

Transcript

  1. Creating a Voice- Driven TV Remote with Azure and Alexa

    Greg Shackles Principal Engineer, Olo @gregshackles.com @gshackles
  2. Project Goals Change channels by searching listings Finally use that

    Raspberry Pi I bought 5 years ago No other servers/infrastructure to manage Use F# and Azure As cheap as possible
  3. Channel Format <?xml version=“1.0” encoding=“UTF-8” ?> <tv date=“12/03/2016” source-info-url=“http://www.tvmedia.ca” source-info-name=“TV

    Media”> <channel id=“1766.stations.xmltv.tvmedia.ca”> <display-name>CBS (WCBS) New York, NY</display-name> <display-name>WCBS-TV1</display-name> <display-name>2</display-name> <url>newyork.cbslocal.com/</url> <icon src=“http://cdn.tvpassport.com/image/station/76x28/cbs.png”/> </channel> </tv>
  4. Parsing the Data type Channels = XmlProvider<"./channels.xml"> let doc =

    Channels.Parse (File.ReadAllText "./channels.xml") type TvChannel = { XmlTvId: string; FullName: string; DisplayName: string; Number: int }
  5. Loading Into SQL doc.Channels |> Array.map (fun channel -> match

    channel.DisplayNames with | [| fullName; displayName; number |] -> Some({ XmlTvId = channel.Id FullName = fullName.String.Value DisplayName = Regex.Replace(displayName.String.Value, "\\-HD$", "", RegexOptions.None) Number = number.Number.Value }) | _ -> None ) |> Array.choose id |> Array.filter (fun channel -> channel.Number > 500 && channel.Number < 1000) |> Array.iter (fun channel -> use cmd = new SqlCommandProvider< "INSERT INTO Channel VALUES (@xmlTvId, @displayName, @fullName, @number)", "name=TVListings">() cmd.Execute(xmlTvId = channel.XmlTvId, displayName = channel.DisplayName, fullName = channel.FullName, number = channel.Number) |> ignore printfn "Added channel: %s" channel.DisplayName )
  6. DownloadLineup open FSharp.Data let Run(timerTrigger: TimerInfo, log: TraceWriter, lineupsBlob: byref<string>)

    = let lineupUrl = sprintf "https://www.xmltvlistings.com/xmltv/get/%s/%s/2/0" (Environment.GetEnvironmentVariable("XmlTvApiKey")) (Environment.GetEnvironmentVariable("XmlTvLineupId")) let xml = Http.RequestString(lineupUrl, responseEncodingOverride = "utf-8") lineupsBlob <- xml sprintf "Successfully downloaded XML, length: %i" xml.Length |> log.Info Runs every morning
  7. ImportLineup type Listings = XmlProvider<"D:\\home\\site\\wwwroot\\ImportLineup\\schedule-sample.xml"> let addShow (log: TraceWriter) (channelLookup:

    IDictionary<string, int>) (show: Listings.Programme) = match channelLookup.TryGetValue show.Channel with | (true, channelId) -> let title = match (show.Title.Value, show.SubTitle) with | "Movie", Some(subtitle) -> subtitle.Value.Trim() | title, _ -> title.Trim() let description = if show.Desc.IsSome then show.Desc.Value.Value else "" Database.addShow title show.Start show.Stop channelId description show.Category.Value sprintf "Added show : %s" title |> log.Info | (false, _) -> () Runs when a new file arrives in blob storage
  8. Azure Search Service Created indexes for channels shows Both sourced

    from SQL Server Free tier: 50MB, 10K documents
  9. Modifying ImportLineup { "type": "queue", "name": "outputQueueItem", "queueName": "rebuild-requests", "connection":

    "tvlistingstorage_STORAGE", "direction": "out" } let Run(xml: string, name: string, log: TraceWriter, outputQueueItem: byref<string>) = // ... outputQueueItem <- name Write to queue when it finishes importing:
  10. RebuildSearchIndex type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq 125", "top":

    100, "search": "seinf type SearchResults = JsonProvider<""" {"value": [{ "ShowId": "abc" }]} """> type DeleteRequest = JsonProvider<""" {"value": [{ "@search.action": "delete", "ShowId": "abc" }] let search query filter count = SearchRequest.Root(filter, count, query).JsonValue.ToString() |> postJson "indexes('shows')/docs/search" |> SearchResults.Parse |> fun results -> results.Value let rec clearIndex() = let results = search "" "" 1000 if not <| Seq.isEmpty results then results |> Array.map (fun result -> DeleteRequest.Value("delete", result.ShowId)) |> fun ops -> DeleteRequest.Root(ops).JsonValue.ToString() |> postJson "indexes('shows')/docs/index" |> ignore clearIndex() let rebuildIndex() = postJson "indexers/shows-indexer/run" "" |> ignore
  11. Exposing the API Registered shackles.house Certificate through Let's Encrypt No-IP

    $35 / year for custom domain support Agent running on the Raspberry Pi Proxy through NGINX
  12. Let's Encrypt certbot-auto certonly -a webroot --webroot-path=/var/www/html -d shackles.house -d

    www.shackles.house 0 3 * * 1 /usr/bin/certbot-auto renew >> /var/log/certbot-renew.log 5 3 * * 1 /etc/init.d/nginx restart Renew the certificate every week:
  13. NGINX Configuration limit_req_zone $binary_remote_addr zone=harmonyzone:10m rate=1r/s; server { listen 9000

    ssl default_server; listen [::]:9000 ssl default_server; # security configuration omitted, see blog for details if ($http_authorization != 'letmein') { return 401; } location / { limit_req zone=harmonyzone burst=5; proxy_pass http://127.0.0.1:8282; proxy_redirect off; } }
  14. DirectCommand Intent { "intents": [ { "intent": "DirectCommand", "slots": [

    { "name": "command", "type": "command_name" }, { "name": "target", "type": "command_target" } ] } ] } DirectCommand {command} DirectCommand {command} the {target} DirectCommand {command} {target} Sample utterances:
  15. RemoteSkill let executeCommand commandSlug = let url = sprintf "%s/commands/%s"

    baseUrl commandSlug let authHeader = "Authorization", apiKey Http.RequestString(url, headers = [authHeader], httpMethod = "POST") |> ignore let handleDirectCommand (intent: Intent) = match (Commands.getCommand intent.Slots.["command"].Value) with | Some(command) -> Commands.executeCommand command.Slug buildResponse "OK" true | None -> buildResponse "Sorry, that command is not available right now" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
  16. RemoteSkill type RemoteSpeechlet(log: TraceWriter) = inherit Speechlet() override this.OnIntent(request: IntentRequest,

    session: Session): SpeechletResponse = sprintf "OnIntent: request %s, session %s" request.RequestId session.SessionId |> log.Info handleIntent request.Intent let Run(req: HttpRequestMessage, log: TraceWriter) = let speechlet = RemoteSpeechlet log speechlet.GetResponse req AlexaSkillsKit: github.com/AreYouFreeBusy/AlexaSkillsKit.NET
  17. Changing the Channel let changeChannel number = string number |>

    Seq.map string |> Seq.iter executeCommand executeCommand "select"
  18. Querying Search Service type SearchRequest = JsonProvider<""" {"filter": "ChannelId eq

    125", "top": 100, "search": "sei type ShowSearchResults = JsonProvider<""" {"value": [{"@search.score": 2.5086286, "ShowId": "10 "StartTime": "20170102200000", "EndTime": "Description": "Yada yada yada", "Categor type ChannelSearchResults = JsonProvider<""" {"value": [{"@search.score": 1, "ChannelId": "150" "DisplayName": "CBSSN-HD", "FullName": let private search index (request: SearchRequest.Root) = let url = sprintf "%s/indexes('%s')/docs/search?api-version=2015-02-28" baseUrl index let apiKeyHeader = "api-key", apiKey let json = request.JsonValue.ToString() Http.RequestString(url, body = TextRequest json, headers = [ ContentType HttpContentTypes.Json; apiKeyHeader ]) let private searchShows request = search "shows" request |> ShowSearchResults.Parse let private searchChannels request = search "channels" request |> ChannelSearchResults.Parse
  19. Finding a Show let findShowOnNow (name: string) = let timestamp

    = DateTime.UtcNow.ToString("yyyyMMddHHmmss") let filter = sprintf "StartTime lt '%s' and EndTime gt '%s'" timestamp timestamp SearchRequest.Root(filter, 1, name) |> searchShows |> fun result -> Array.tryPick Some result.Value
  20. WatchShow Intent { "intent": "WatchShow", "slots": [ { "name": "name",

    "type": "show_name" } ] } WatchShow turn on {name} WatchShow watch {name} WatchShow switch to {name} Sample utterances:
  21. RemoteSkill let handleWatchShow (intent: Intent) = Search.findShowOnNow intent.Slots.["name"].Value |> function

    | Some(show) -> Search.findChannel show.ChannelId |> function | Some(channel) -> Commands.changeChannel channel buildResponse "OK" true | None -> buildResponse "Sorry, I could not find the channel for that show" true | None -> buildResponse "Sorry, I could not find that show" true let handleIntent (intent: Intent) = match intent.Name with | "DirectCommand" -> handleDirectCommand intent | "WatchShow" -> handleWatchShow intent | _ -> buildResponse "Sorry, I'm not sure how to do that" true
  22. The HTTP API Had Some Downsides Exposed a public API

    to my living room Performance kind of sucked A lot of moving parts No-IP NGINX Let's Encrypt
  23. MQTT Message Queue Telemetry Transport Lightweight publish/subscribe protocol Built on

    TCP protocol Commonly used in IoT scenarios Supported by Azure IoT Hub
  24. IoT Hub Bridge const mqttServer = require('net').createServer(broker.handle); mqttServer.listen(1883, () =>

    console.log('MQTT broker listening')); function onAzureConnect(err) { azureClient.on('error', console.error); azureClient.on('disconnect', function () { azureClient.removeAllListeners(); azureClient.open(onAzureConnect); }); function sendCommand(command) { const [topic, payload] = command.split(';'); broker.publish({ topic, payload }); } azureClient.on('message', msg => msg.data.toString() .split('\n') .forEach((command, index) => setTimeout( () => sendCommand(command), index * CommandDelayMs))); } azureClient.open(onAzureConnect);
  25. RemoteSkill let executeCommands commandSlugs = async { commandSlugs |> Seq.map

    (fun slug -> sprintf "harmony-api/hubs/living-room/command;%s" slug) |> String.concat "\n" |> Encoding.ASCII.GetBytes |> fun bytes -> new Message(bytes) |> fun message -> serviceClient.SendAsync("harmony-bridge", message) |> Async.AwaitIAsyncResult |> Async.Ignore |> ignore } |> Async.RunSynchronously let executeCommand commandSlug = executeCommands [commandSlug]
  26. Persisting Available Commands CREATE TABLE AvailableCommand( AvailableCommandId int IDENTITY(1,1) NOT

    NULL, Name nvarchar(32) NOT NULL, Slug nvarchar(16) NOT NULL, Label nvarchar(32) NOT NULL ) broker.on('publish', packet => { if (packet.topic !== 'harmony-api/hubs/living-room/current_activity') { return; } request({ url: 'http://localhost:8282/hubs/living-room/commands', json: true }) .then(res => updateAvailableCommands(res.commands)); });
  27. Timing Sub-Operations let private telemetryClient = TelemetryClient(InstrumentationKey = instrumentationKey) let

    setOperationId operationId = telemetryClient.Context.Operation.Id <- operationId let startOperation (name:string) = telemetryClient.StartOperation<DependencyTelemetry>(name) let private searchShows request = use operation = Telemetry.startOperation "ShowSearch" search "shows" request |> ShowSearchResults.Parse let private searchChannels request = use operation = Telemetry.startOperation "ChannelSearch" search "channels" request |> ChannelSearchResults.Parse let executeCommand commandSlug = use operation = Telemetry.startOperation "ExecuteCommand" // ...
  28. Total: $7.56 Costs Component Monthly Cost XML TV Listings $1.67

    SQL Server $4.99 Storage $0.63 App Service $0.27 IoT Hub Free Search Service Free Application Insights Free