Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Data Products at Home: Automating Philips Hue Lights

Building Data Products at Home: Automating Philips Hue Lights

Today’s smart devices come with full fledged API’s driving the proliferation of micro-services as apps. Philips Hue lights are no exception. With the coming of age of open source libraries for data science, building data products at home has become easier than ever. Not only can we automate away repetitive task , we also get to experiment freely in the tools we use; a comfortable way to learn new tricks.

These are the slides of the talk I gave at the 2017 Budapest BI Conference on how I built a small system to control my Philips Hue lights at home, and what I have learned in the process. The brains of the project is written in R, while more general programming tasks such as interfacing with API’s and workflow management are done in Python.

Tamas Szilagyi

November 20, 2017
Tweet

More Decks by Tamas Szilagyi

Other Decks in Programming

Transcript

  1. Building data products at
    home
    Self-learning lights
    tamaszilagyi.com

    View full-size slide

  2. • Senior Data Analyst at in Amsterdam
    • Avid user
    • Blog on hobby data projects: http://tamaszilagyi.com/
    tamaszilagyi.com
    About me

    View full-size slide

  3. What are data
    products?
    “Data products automate complex analysis tasks or use technology to expand
    the utility of a data informed model, algorithm or inference.”
    (Coursera: Developing Data Products)
    tamaszilagyi.com

    View full-size slide

  4. • You most likely developed / worked on data products.
    tamaszilagyi.com
    What are data
    products?

    View full-size slide

  5. • You most likely developed / worked on data products.
    • 100% sure interacted with them as an end-user
    tamaszilagyi.com
    What are data
    products?

    View full-size slide

  6. Doing them at home
    Pro’s
    • Both creator and end-user
    tamaszilagyi.com

    View full-size slide

  7. Doing them at home
    Pro’s
    • Both creator and end-user
    • Experiment with tools
    tamaszilagyi.com

    View full-size slide

  8. Doing them at home
    Pro’s
    • Both creator and end-user
    • Experiment with tools
    • No deadline
    tamaszilagyi.com

    View full-size slide

  9. Doing them at home
    Pro’s
    • Both creator and end-user
    • Experiment with tools
    • No deadline
    Cons?
    tamaszilagyi.com

    View full-size slide

  10. Doing them at home
    Pro’s
    • Both creator and end-user
    • Experiment with tools
    • No deadline
    Cons?
    tamaszilagyi.com

    View full-size slide

  11. What do we need?
    Raspberry Pi Anything with an API R & Python
    tamaszilagyi.com

    View full-size slide

  12. R and Python
    …wait, what?
    tamaszilagyi.com

    View full-size slide

  13. Most discussion focuses on which one is better.
    Why not use both?
    Complement each other.
    tamaszilagyi.com
    R and Python
    …wait, what?

    View full-size slide

  14. Case study: Hue lights
    Automating away the light switch
    tamaszilagyi.com

    View full-size slide

  15. Business as usual
    tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch

    View full-size slide

  16. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch

    View full-size slide

  17. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    • Merge daily files

    • Train model
    Save .json daily
    Expose model as API PUT request
    GET request

    View full-size slide

  18. 1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge
    tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch

    View full-size slide

  19. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge

    View full-size slide

  20. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge

    View full-size slide

  21. Imagine a world…
    tamaszilagyi.com

    View full-size slide

  22. • Prototype on laptop
    • Push to git
    • Deploy on Pi
    tamaszilagyi.com
    Workflow

    View full-size slide

  23. • Python is more mature when it comes to general purpose programming
    • Glue is workflow management tool Luigi
    • Task are executed in order of dependencies
    tamaszilagyi.com
    Log, clean and save data

    View full-size slide

  24. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.

    View full-size slide

  25. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.
    2. Copy data to S3 for long-term storage.

    View full-size slide

  26. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.
    2. Copy data to S3 for long-term storage.
    3. Read full data, train and save model.

    View full-size slide

  27. Log, clean and save data
    An example of a Luigi task
    tamaszilagyi.com

    View full-size slide

  28. This way we can chain together all the tasks of our pipeline.
    One task’s output goes into the next one’s require() method.
    tamaszilagyi.com
    Log, clean and save data
    An example of a Luigi task

    View full-size slide

  29. If you wait long enough,
    DAG’s become trippy
    tamaszilagyi.com

    View full-size slide

  30. This is where R really shines:
    • Interactive use • Stats
    tamaszilagyi.com
    Explore data, train model

    View full-size slide

  31. tidyr, dplyr -> ggplot2 -> caret (-> jug)
    tamaszilagyi.com
    Explore data, train model

    View full-size slide

  32. tamaszilagyi.com
    Explore data, train model
    Read from AWS

    View full-size slide

  33. tamaszilagyi.com
    Explore data, train model
    Data is quite messy

    View full-size slide

  34. tamaszilagyi.com
    Explore data, train model
    dplyr & tidyr to the rescue

    View full-size slide

  35. tamaszilagyi.com
    Explore data, train model
    dplyr & tidyr to the rescue

    View full-size slide

  36. tamaszilagyi.com
    Explore data, train model
    Bin brightness, I want to classify

    View full-size slide

  37. tamaszilagyi.com
    Add time based features
    Explore data, train model

    View full-size slide

  38. tamaszilagyi.com
    • Unbalanced dataset
    • We have a time series
    Potential caveats
    Explore data, train model

    View full-size slide

  39. tamaszilagyi.com
    Caret has solutions
    Explore data, train model

    View full-size slide

  40. tamaszilagyi.com
    Caret has solutions
    Explore data, train model

    View full-size slide

  41. Performance
    tamaszilagyi.com
    Explore data, train model

    View full-size slide

  42. Add model training as task
    to Luigi pipeline?
    Package up R code as a script and use subprocess.

    View full-size slide

  43. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3

    View full-size slide

  44. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3
    2. Expose model as an API on Pi

    View full-size slide

  45. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3
    2. Expose model as an API on Pi
    3. GET prediction, PUT brightness value

    View full-size slide

  46. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function

    View full-size slide

  47. Communicate with Bridge
    tamaszilagyi.com
    And, that’s it.

    View full-size slide

  48. tamaszilagyi.com
    Do we need an interface?
    App > Chatbot > Script in Terminal

    View full-size slide

  49. Reaping the benefits
    tamaszilagyi.com

    View full-size slide

  50. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying

    View full-size slide

  51. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying
    • Try out new tech on something else that the iris dataset

    View full-size slide

  52. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying
    • Try out new tech on something else that the iris dataset
    • Compiling Rcpp on Raspberry Pi takes forever

    View full-size slide

  53. tamaszilagyi.com
    Questions?
    Keep in touch!
    http://tamaszilagyi.com/
    @tudosgar
    https://www.linkedin.com/in/tszilagyi/

    View full-size slide

  54. tamaszilagyi.com
    Thank you!
    Now, time for drinks.

    View full-size slide