Building Data Products at Home: Automating Philips Hue Lights

Today’s smart devices come with full fledged API’s driving the proliferation of micro-services as apps. Philips Hue lights are no exception. With the coming of age of open source libraries for data science, building data products at home has become easier than ever. Not only can we automate away repetitive task , we also get to experiment freely in the tools we use; a comfortable way to learn new tricks.

These are the slides of the talk I gave at the 2017 Budapest BI Conference on how I built a small system to control my Philips Hue lights at home, and what I have learned in the process. The brains of the project is written in R, while more general programming tasks such as interfacing with API’s and workflow management are done in Python.

Transcript

1. Building data products at
   home
   Self-learning lights
   tamaszilagyi.com
   

   View Slide

2. • Senior Data Analyst at in Amsterdam
   • Avid user
   • Blog on hobby data projects: http://tamaszilagyi.com/
   tamaszilagyi.com
   About me
   

   View Slide

3. What are data
   products?
   “Data products automate complex analysis tasks or use technology to expand
   the utility of a data informed model, algorithm or inference.”
   (Coursera: Developing Data Products)
   tamaszilagyi.com
   

   View Slide

4. • You most likely developed / worked on data products.
   tamaszilagyi.com
   What are data
   products?
   

   View Slide

5. • You most likely developed / worked on data products.
   • 100% sure interacted with them as an end-user
   tamaszilagyi.com
   What are data
   products?
   

   View Slide

6. Doing them at home
   Pro’s
   • Both creator and end-user
   tamaszilagyi.com
   

   View Slide

7. Doing them at home
   Pro’s
   • Both creator and end-user
   • Experiment with tools
   tamaszilagyi.com
   

   View Slide

8. Doing them at home
   Pro’s
   • Both creator and end-user
   • Experiment with tools
   • No deadline
   tamaszilagyi.com
   

   View Slide

9. Doing them at home
   Pro’s
   • Both creator and end-user
   • Experiment with tools
   • No deadline
   Cons?
   tamaszilagyi.com
   

   View Slide

10. Doing them at home
    Pro’s
    • Both creator and end-user
    • Experiment with tools
    • No deadline
    Cons?
    tamaszilagyi.com
    

    View Slide

11. What do we need?
    Raspberry Pi Anything with an API R & Python
    tamaszilagyi.com
    

    View Slide

12. R and Python
    …wait, what?
    tamaszilagyi.com
    

    View Slide

13. Most discussion focuses on which one is better.
    Why not use both?
    Complement each other.
    tamaszilagyi.com
    R and Python
    …wait, what?
    

    View Slide

14. Case study: Hue lights
    Automating away the light switch
    tamaszilagyi.com
    

    View Slide

15. Business as usual
    tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    

    View Slide

16. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    

    View Slide

17. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    • Merge daily files

    • Train model
    Save .json daily
    Expose model as API PUT request
    GET request
    

    View Slide

18. 1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge
    tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    

    View Slide

19. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge
    

    View Slide

20. tamaszilagyi.com
    Case study: Hue lights
    Automating away the light switch
    1. Log, clean and save data
    2. Explore, train model
    3. Communicate with bridge
    

    View Slide

21. Imagine a world…
    tamaszilagyi.com
    

    View Slide

22. • Prototype on laptop
    • Push to git
    • Deploy on Pi
    tamaszilagyi.com
    Workflow
    

    View Slide

23. • Python is more mature when it comes to general purpose programming
    • Glue is workflow management tool Luigi
    • Task are executed in order of dependencies
    tamaszilagyi.com
    Log, clean and save data
    

    View Slide

24. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.
    

    View Slide

25. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.
    2. Copy data to S3 for long-term storage.
    

    View Slide

26. tamaszilagyi.com
    Log, clean and save data
    1. Send GET request and save file to new file every day locally.
    2. Copy data to S3 for long-term storage.
    3. Read full data, train and save model.
    

    View Slide

27. Log, clean and save data
    An example of a Luigi task
    tamaszilagyi.com
    

    View Slide

28. This way we can chain together all the tasks of our pipeline.
    One task’s output goes into the next one’s require() method.
    tamaszilagyi.com
    Log, clean and save data
    An example of a Luigi task
    

    View Slide

29. If you wait long enough,
    DAG’s become trippy
    tamaszilagyi.com
    

    View Slide

30. This is where R really shines:
    • Interactive use • Stats
    tamaszilagyi.com
    Explore data, train model
    

    View Slide

31. tidyr, dplyr -> ggplot2 -> caret (-> jug)
    tamaszilagyi.com
    Explore data, train model
    

    View Slide

32. tamaszilagyi.com
    Explore data, train model
    Read from AWS
    

    View Slide

33. tamaszilagyi.com
    Explore data, train model
    Data is quite messy
    

    View Slide

34. tamaszilagyi.com
    Explore data, train model
    dplyr & tidyr to the rescue
    

    View Slide

35. tamaszilagyi.com
    Explore data, train model
    dplyr & tidyr to the rescue
    

    View Slide

36. tamaszilagyi.com
    Explore data, train model
    Bin brightness, I want to classify
    

    View Slide

37. tamaszilagyi.com
    Add time based features
    Explore data, train model
    

    View Slide

38. tamaszilagyi.com
    • Unbalanced dataset
    • We have a time series
    Potential caveats
    Explore data, train model
    

    View Slide

39. tamaszilagyi.com
    Caret has solutions
    Explore data, train model
    

    View Slide

40. tamaszilagyi.com
    Caret has solutions
    Explore data, train model
    

    View Slide

41. Performance
    tamaszilagyi.com
    Explore data, train model
    

    View Slide

42. Add model training as task
    to Luigi pipeline?
    Package up R code as a script and use subprocess.
    

    View Slide

43. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3
    

    View Slide

44. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3
    2. Expose model as an API on Pi
    

    View Slide

45. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    1. Read last model from S3
    2. Expose model as an API on Pi
    3. GET prediction, PUT brightness value
    

    View Slide

46. Communicate with Bridge
    tamaszilagyi.com
    Custom predict function
    

    View Slide

47. Communicate with Bridge
    tamaszilagyi.com
    And, that’s it.
    

    View Slide

48. tamaszilagyi.com
    Do we need an interface?
    App > Chatbot > Script in Terminal
    

    View Slide

49. Reaping the benefits
    tamaszilagyi.com
    

    View Slide

50. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying
    

    View Slide

51. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying
    • Try out new tech on something else that the iris dataset
    

    View Slide

52. tamaszilagyi.com
    Lessons learned
    • Seeing the model in action has something deeply satisfying
    • Try out new tech on something else that the iris dataset
    • Compiling Rcpp on Raspberry Pi takes forever
    

    View Slide

53. tamaszilagyi.com
    Questions?
    Keep in touch!
    http://tamaszilagyi.com/
    @tudosgar
    https://www.linkedin.com/in/tszilagyi/
    

    View Slide

54. tamaszilagyi.com
    Thank you!
    Now, time for drinks.
    

    View Slide