Django for Internet of Things: from hackathon to production

talk at DjangoCon Europe 2016
code at https://github.com/aschn/cookiecutter-django-iot

Transcript

1. Django for Internet of Things: from hackathon to production Anna Schneider DjangoCon Europe, 31 March 2016 @windupanna
2. Hello! I’m Anna :) • learned Python during PhD in biophysics • learned Django at hackathon where WattTime started • now co-founder and CTO at WattTime (shoutout to UnconsciousBiasProject.org @UBP_STEM)
3. what?? Django for Internet of Things: from hackathon to production
4. everyone’s favorite framework Django
5. Internet of Things (IoT) when a thing you don’t normally think of as a computer can transmit data and respond to controls in real time ⏰
6. some people like to write the code on the thing, I like to talk to things through their APIs Internet of Things (IoT) ☁️ ☁️ you them the thing
7. a really fun way to write really fragile code really quickly hackathon ☕️⚠️
8. when you can ship your code and trust it while you’re having fun at DjangoCon production
9. design patterns (and anti-patterns) for writing and deploying Django projects to monitor and control an IoT device using its API so you can get started fast then build well what you’ll learn ☕️
10. python manage.py startproject awesome_iot_hackathon ☕️⚠️
11. models!
12. the books app Book title author year Book title author year Author name hometown Book title author year Book title author year Author name hometown Author name hometown
13. the books IoT app Observation value device timestamp Device name location Device name location Device name location Observation value device timestamp Observation value device timestamp Observation value device timestamp
14. the IoT app Observation value device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID Observation value device timestamp Observation value device timestamp Observation value device timestamp ☕️ vendor’s unique ID(s) whatever data you need to send to their API
15. Observation value device timestamp Observation value device timestamp Observation value device timestamp the IoT app Observation value device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID use db_index you’ll be sorting on this a lot! Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp
16. Observation value device timestamp Observation value device timestamp Observation value device timestamp the IoT app Attribute value units device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID Attribute vs Status numerical vs str/bool values Observation value device timestamp Observation value device timestamp Observation value device timestamp Attribute value units device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Attribute value units device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Status is_on device timestamp ☕️
17. views? models
18. views? tasks!
19. views? tasks! outside of request/response cycle ☁️ ☁️ you them the thing request response
20. what tasks do we need to do? pull new attributes pull new on/off statuses set attributes set on/off statuses monitor control do something awesome generalizable custom
21. tasks.py def do_something_awesome(device_id): # get some cool data thing_to_do = cool_data_source() # do something awesome awesome_action(device_id, thing_to_do) ☕️
22. tasks.py def pull_status(device_id): # get device from pk device = Device.objects.get( pk=device_id ) # get status from device client is_on = client.get_status(device) # create status status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), ) # return return [status.pk] pass pks pro: don’t rely on database state con: may be extra queries
23. tasks.py def pull_status(device_id): # get device from pk device = Device.objects.get( pk=device_id ) # get status from device client is_on = client.get_status(device) # create status status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), ) # return return [status.pk] encapsulate client clean! swappable!
24. put it together! models tasks
25. the hackathon app myapp models.py Device Attribute Status tasks.py pull_status set_status pull_attributes set_attributes client.py admin.py views.py tests.py ☕️
26. the production apps devices models.py admin.py views.py tests.py interactions tasks.py views.py tests.py vendor client.py tests.py observations models.py admin.py views.py tests.py views for adding/removing devices analytics, DRF for dashboards models for logging, views for clickable tasks swappable vendors
27. why not tasks as Device model methods? devices models.py admin.py views.py tests.py observations models.py admin.py views.py tests.py tightly coupled not supported by celery core http://docs.celeryproject.org/en/latest/ reference/celery.contrib.methods.html device.pull_status() vendor client.py tests.py
28. devices models.py admin.py views.py tests.py interactions tasks.py views.py tests.py observations models.py admin.py views.py tests.py why not client as Device model methods? works ok, but less reusable and less scalable device.send_status_to_vendor()
29. deploy + automate! models tasks apps
30. deploy + automate! goals: • run any task (control or monitor) • at frequent, deterministic times • outside of request/response cycle cron in the cloud
31. 1) the hackathon way two ways to automate management commands + Heroku Scheduler ☕️ python manage.py interact pull_status --device_id=1
32. Heroku Scheduler pros: • easy to set up cons: • limited frequencies (daily, hourly, 10 min) • “best effort” service (may skip some) ☕️
33. 2) the production way two ways to automate task functions + celery periodic task scheduler
34. Celery distributed message queuing system for asynchronous stuff slow event-driven tasks send the user sign-up email scheduled periodic tasks run the daily report
35. web servers Celery architecture (how) tasks tasks worker servers messages results result store message broker transport queue messages tasks scheduler
36. message broker transport queue or or Django ORM (djcelery) web servers worker servers messages tasks tasks messages results result store or or Django ORM (djcelery) or None tasks scheduler Celery architecture (how)
37. tasks.py from celery import shared_task @shared_task def pull_status(device_id): ... @shared_task def refresh_all(): ... just add decorator :) (what)
38. from celery.schedules import crontab SCHEDULE = { 'run_task': { # use the actual path 'task': 'path.to.task', # every 5 minutes 'schedule': crontab(minute='*/5'), # args and kwargs 'args': [], 'kwargs': {}, }, } schedule.py very close to crontab once per minute - once per year (when)
39. from celery.schedules import crontab SCHEDULE = { 'run_task': { # use the actual path 'task': 'path.to.task', # every 5 minutes 'schedule': crontab(minute='*/5'), # args and kwargs 'args': [], 'kwargs': {'device_id': d.pk}, } for d in Device.objects.all() } schedule.py static arguments only code is only evaluated once at run time, better to have one task spawn daughters (when)
40. # set the default Django settings module os.environ.setdefault( 'DJANGO_SETTINGS_MODULE', 'django_iot.settings.production' ) app = Celery('django_iot') # configure app.config_from_object('django.conf:settings') # autodiscover tasks in any app app.autodiscover_tasks(settings.INSTALLED_APPS) # set schedule from django_iot.apps.interactions.schedule import SCHEDULE app.conf.CELERYBEAT_SCHEDULE = SCHEDULE boilerplate like wsgi.py copy from http://docs.celeryproject.org/ en/latest/django/first-steps-with- django.html celery.py swappable and composable schedules
41. from __future__ import absolute_import from .celery import app as celery_app make celery app available to @shared_task, and to Procfile __init__.py web: gunicorn django_iot.wsgi --log-file - scheduler: celery worker -B -A django_iot worker: celery worker -A django_iot Procfile -B for beat worker and scheduler in one! warning!! with free Heroku dynos, scheduler sleeps if web dyno is inactive for 30 min
42. production, Celerified devices models.py admin.py views.py tests.py interactions tasks.py schedule.py views.py tests.py brand client.py tests.py observations models.py admin.py views.py tests.py django_iot __init__.py celery.py settings.py wsgi.py urls.py django_iot Procfile requirements.txt manage.py
43. cookiecutter https://github.com/aschn/cookiecutter-django-iot.git ☕️
44. • data model: Device, Attribute, Status • tasks not views: outside request/response • ☕️ easy but rigid: Heroku Scheduler • flexible but complex: celery periodic tasks • hack better+faster: cookiecutter-django-iot • IoT can be for good, not just for fun and profit what have we learned?
45. Thanks!! Anna Schneider @windupanna @wattTime cookiecutter https://github.com/aschn/cookiecutter-django-iot.git