Building Data Products at Home: Automating Philips Hue Lights

Building Data Products at Home: Automating Philips Hue Lights

Today’s smart devices come with full fledged API’s driving the proliferation of micro-services as apps. Philips Hue lights are no exception. With the coming of age of open source libraries for data science, building data products at home has become easier than ever. Not only can we automate away repetitive task , we also get to experiment freely in the tools we use; a comfortable way to learn new tricks.

These are the slides of the talk I gave at the 2017 Budapest BI Conference on how I built a small system to control my Philips Hue lights at home, and what I have learned in the process. The brains of the project is written in R, while more general programming tasks such as interfacing with API’s and workflow management are done in Python.

Fa8c8d5560f95f32cb78e164ba2d222e?s=128

Tamas Szilagyi

November 20, 2017
Tweet

Transcript

  1. Building data products at home Self-learning lights tamaszilagyi.com

  2. • Senior Data Analyst at in Amsterdam • Avid user

    • Blog on hobby data projects: http://tamaszilagyi.com/ tamaszilagyi.com About me
  3. What are data products? “Data products automate complex analysis tasks

    or use technology to expand the utility of a data informed model, algorithm or inference.” (Coursera: Developing Data Products) tamaszilagyi.com
  4. • You most likely developed / worked on data products.

    tamaszilagyi.com What are data products?
  5. • You most likely developed / worked on data products.

    • 100% sure interacted with them as an end-user tamaszilagyi.com What are data products?
  6. Doing them at home Pro’s • Both creator and end-user

    tamaszilagyi.com
  7. Doing them at home Pro’s • Both creator and end-user

    • Experiment with tools tamaszilagyi.com
  8. Doing them at home Pro’s • Both creator and end-user

    • Experiment with tools • No deadline tamaszilagyi.com
  9. Doing them at home Pro’s • Both creator and end-user

    • Experiment with tools • No deadline Cons? tamaszilagyi.com
  10. Doing them at home Pro’s • Both creator and end-user

    • Experiment with tools • No deadline Cons? tamaszilagyi.com
  11. What do we need? Raspberry Pi Anything with an API

    R & Python tamaszilagyi.com
  12. R and Python …wait, what? tamaszilagyi.com

  13. Most discussion focuses on which one is better. Why not

    use both? Complement each other. tamaszilagyi.com R and Python …wait, what?
  14. Case study: Hue lights Automating away the light switch tamaszilagyi.com

  15. Business as usual tamaszilagyi.com Case study: Hue lights Automating away

    the light switch
  16. tamaszilagyi.com Case study: Hue lights Automating away the light switch

  17. tamaszilagyi.com Case study: Hue lights Automating away the light switch

    • Merge daily files
 • Train model Save .json daily Expose model as API PUT request GET request
  18. 1. Log, clean and save data 2. Explore, train model

    3. Communicate with bridge tamaszilagyi.com Case study: Hue lights Automating away the light switch
  19. tamaszilagyi.com Case study: Hue lights Automating away the light switch

    1. Log, clean and save data 2. Explore, train model 3. Communicate with bridge
  20. tamaszilagyi.com Case study: Hue lights Automating away the light switch

    1. Log, clean and save data 2. Explore, train model 3. Communicate with bridge
  21. Imagine a world… tamaszilagyi.com

  22. • Prototype on laptop • Push to git • Deploy

    on Pi tamaszilagyi.com Workflow
  23. • Python is more mature when it comes to general

    purpose programming • Glue is workflow management tool Luigi • Task are executed in order of dependencies tamaszilagyi.com Log, clean and save data
  24. tamaszilagyi.com Log, clean and save data 1. Send GET request

    and save file to new file every day locally.
  25. tamaszilagyi.com Log, clean and save data 1. Send GET request

    and save file to new file every day locally. 2. Copy data to S3 for long-term storage.
  26. tamaszilagyi.com Log, clean and save data 1. Send GET request

    and save file to new file every day locally. 2. Copy data to S3 for long-term storage. 3. Read full data, train and save model.
  27. Log, clean and save data An example of a Luigi

    task tamaszilagyi.com
  28. This way we can chain together all the tasks of

    our pipeline. One task’s output goes into the next one’s require() method. tamaszilagyi.com Log, clean and save data An example of a Luigi task
  29. If you wait long enough, DAG’s become trippy tamaszilagyi.com

  30. This is where R really shines: • Interactive use •

    Stats tamaszilagyi.com Explore data, train model
  31. tidyr, dplyr -> ggplot2 -> caret (-> jug) tamaszilagyi.com Explore

    data, train model
  32. tamaszilagyi.com Explore data, train model Read from AWS

  33. tamaszilagyi.com Explore data, train model Data is quite messy

  34. tamaszilagyi.com Explore data, train model dplyr & tidyr to the

    rescue
  35. tamaszilagyi.com Explore data, train model dplyr & tidyr to the

    rescue
  36. tamaszilagyi.com Explore data, train model Bin brightness, I want to

    classify
  37. tamaszilagyi.com Add time based features Explore data, train model

  38. tamaszilagyi.com • Unbalanced dataset • We have a time series

    Potential caveats Explore data, train model
  39. tamaszilagyi.com Caret has solutions Explore data, train model

  40. tamaszilagyi.com Caret has solutions Explore data, train model

  41. Performance tamaszilagyi.com Explore data, train model

  42. Add model training as task to Luigi pipeline? Package up

    R code as a script and use subprocess.
  43. Communicate with Bridge tamaszilagyi.com Custom predict function 1. Read last

    model from S3
  44. Communicate with Bridge tamaszilagyi.com Custom predict function 1. Read last

    model from S3 2. Expose model as an API on Pi
  45. Communicate with Bridge tamaszilagyi.com Custom predict function 1. Read last

    model from S3 2. Expose model as an API on Pi 3. GET prediction, PUT brightness value
  46. Communicate with Bridge tamaszilagyi.com Custom predict function

  47. Communicate with Bridge tamaszilagyi.com And, that’s it.

  48. tamaszilagyi.com Do we need an interface? App > Chatbot >

    Script in Terminal
  49. Reaping the benefits tamaszilagyi.com

  50. tamaszilagyi.com Lessons learned • Seeing the model in action has

    something deeply satisfying
  51. tamaszilagyi.com Lessons learned • Seeing the model in action has

    something deeply satisfying • Try out new tech on something else that the iris dataset
  52. tamaszilagyi.com Lessons learned • Seeing the model in action has

    something deeply satisfying • Try out new tech on something else that the iris dataset • Compiling Rcpp on Raspberry Pi takes forever
  53. tamaszilagyi.com Questions? Keep in touch! http://tamaszilagyi.com/ @tudosgar https://www.linkedin.com/in/tszilagyi/

  54. tamaszilagyi.com Thank you! Now, time for drinks.