Django for Internet of Things: from hackathon to production Anna Schneider DjangoCon Europe, 31 March 2016 @windupanna 

Hello! I’m Anna :) • learned Python during PhD in biophysics • learned Django at hackathon where WattTime started • now co-founder and CTO at WattTime (shoutout to UnconsciousBiasProject.org @UBP_STEM) 

what?? Django for Internet of Things: from hackathon to production 

everyone’s favorite framework Django 

Internet of Things (IoT) when a thing you don’t normally think of as a computer can transmit data and respond to controls in real time ⏰ 

some people like to write the code on the thing, I like to talk to things through their APIs Internet of Things (IoT) ☁️ ☁️ you them the thing 

a really fun way to write really fragile code really quickly hackathon ☕️⚠️ 

when you can ship your code and trust it while you’re having fun at DjangoCon production 

design patterns (and anti-patterns) for writing and deploying Django projects to monitor and control an IoT device using its API so you can get started fast then build well what you’ll learn ☕️ 

python manage.py startproject awesome_iot_hackathon ☕️⚠️ 

models! 

the books app Book title author year Book title author year Author name hometown Book title author year Book title author year Author name hometown Author name hometown 

the books IoT app Observation value device timestamp Device name location Device name location Device name location Observation value device timestamp Observation value device timestamp Observation value device timestamp 

the IoT app Observation value device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID Observation value device timestamp Observation value device timestamp Observation value device timestamp ☕️ vendor’s unique ID(s) whatever data you need to send to their API 

Observation value device timestamp Observation value device timestamp Observation value device timestamp the IoT app Observation value device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID use db_index you’ll be sorting on this a lot! Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp 

Observation value device timestamp Observation value device timestamp Observation value device timestamp the IoT app Attribute value units device timestamp Device name location vendor ID Device name location vendor ID Device name location vendor ID Attribute vs Status numerical vs str/bool values Observation value device timestamp Observation value device timestamp Observation value device timestamp Attribute value units device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Attribute value units device timestamp Observation value device timestamp Observation value device timestamp Observation value device timestamp Status is_on device timestamp ☕️ 

views? models 

views? tasks! 

views? tasks! outside of request/response cycle ☁️ ☁️ you them the thing request response 

what tasks do we need to do? pull new attributes pull new on/off statuses set attributes set on/off statuses monitor control do something awesome generalizable custom 

tasks.py def do_something_awesome(device_id): # get some cool data thing_to_do = cool_data_source() # do something awesome awesome_action(device_id, thing_to_do) ☕️ 

tasks.py def pull_status(device_id): # get device from pk device = Device.objects.get( pk=device_id ) # get status from device client is_on = client.get_status(device) # create status status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), ) # return return [status.pk] pass pks pro: don’t rely on database state con: may be extra queries 

tasks.py def pull_status(device_id): # get device from pk device = Device.objects.get( pk=device_id ) # get status from device client is_on = client.get_status(device) # create status status = device.status_set.create( is_on=is_on, valid_at=timezone.now(), ) # return return [status.pk] encapsulate client clean! swappable! 

put it together! models tasks 

the hackathon app myapp models.py Device Attribute Status tasks.py pull_status set_status pull_attributes set_attributes client.py admin.py views.py tests.py ☕️ 

the production apps devices models.py admin.py views.py tests.py interactions tasks.py views.py tests.py vendor client.py tests.py observations models.py admin.py views.py tests.py views for adding/removing devices analytics, DRF for dashboards models for logging, views for clickable tasks swappable vendors 

why not tasks as Device model methods? devices models.py admin.py views.py tests.py observations models.py admin.py views.py tests.py tightly coupled not supported by celery core http://docs.celeryproject.org/en/latest/ reference/celery.contrib.methods.html device.pull_status() vendor client.py tests.py 

devices models.py admin.py views.py tests.py interactions tasks.py views.py tests.py observations models.py admin.py views.py tests.py why not client as Device model methods? works ok, but less reusable and less scalable device.send_status_to_vendor() 

deploy + automate! models tasks apps 

deploy + automate! goals: • run any task (control or monitor) • at frequent, deterministic times • outside of request/response cycle cron in the cloud 

1) the hackathon way two ways to automate management commands + Heroku Scheduler ☕️ python manage.py interact pull_status --device_id=1 

Heroku Scheduler pros: • easy to set up cons: • limited frequencies (daily, hourly, 10 min) • “best effort” service (may skip some) ☕️ 

2) the production way two ways to automate task functions + celery periodic task scheduler 

Celery distributed message queuing system for asynchronous stuff slow event-driven tasks send the user sign-up email scheduled periodic tasks run the daily report 

web servers Celery architecture (how) tasks tasks worker servers messages results result store message broker transport queue messages tasks scheduler 

message broker transport queue or or Django ORM (djcelery) web servers worker servers messages tasks tasks messages results result store or or Django ORM (djcelery) or None tasks scheduler Celery architecture (how) 

tasks.py from celery import shared_task @shared_task def pull_status(device_id): ... @shared_task def refresh_all(): ... just add decorator :) (what) 

from celery.schedules import crontab SCHEDULE = { 'run_task': { # use the actual path 'task': 'path.to.task', # every 5 minutes 'schedule': crontab(minute='*/5'), # args and kwargs 'args': [], 'kwargs': {}, }, } schedule.py very close to crontab once per minute - once per year (when) 

from celery.schedules import crontab SCHEDULE = { 'run_task': { # use the actual path 'task': 'path.to.task', # every 5 minutes 'schedule': crontab(minute='*/5'), # args and kwargs 'args': [], 'kwargs': {'device_id': d.pk}, } for d in Device.objects.all() } schedule.py static arguments only code is only evaluated once at run time, better to have one task spawn daughters (when) 

# set the default Django settings module os.environ.setdefault( 'DJANGO_SETTINGS_MODULE', 'django_iot.settings.production' ) app = Celery('django_iot') # configure app.config_from_object('django.conf:settings') # autodiscover tasks in any app app.autodiscover_tasks(settings.INSTALLED_APPS) # set schedule from django_iot.apps.interactions.schedule import SCHEDULE app.conf.CELERYBEAT_SCHEDULE = SCHEDULE boilerplate like wsgi.py copy from http://docs.celeryproject.org/ en/latest/django/first-steps-with- django.html celery.py swappable and composable schedules 

from __future__ import absolute_import from .celery import app as celery_app make celery app available to @shared_task, and to Procfile __init__.py web: gunicorn django_iot.wsgi --log-file - scheduler: celery worker -B -A django_iot worker: celery worker -A django_iot Procfile -B for beat worker and scheduler in one! warning!! with free Heroku dynos, scheduler sleeps if web dyno is inactive for 30 min 

production, Celerified devices models.py admin.py views.py tests.py interactions tasks.py schedule.py views.py tests.py brand client.py tests.py observations models.py admin.py views.py tests.py django_iot __init__.py celery.py settings.py wsgi.py urls.py django_iot Procfile requirements.txt manage.py 

cookiecutter https://github.com/aschn/cookiecutter-django-iot.git ☕️ 

• data model: Device, Attribute, Status • tasks not views: outside request/response • ☕️ easy but rigid: Heroku Scheduler • flexible but complex: celery periodic tasks • hack better+faster: cookiecutter-django-iot • IoT can be for good, not just for fun and profit what have we learned? 

Thanks!! Anna Schneider @windupanna @wattTime cookiecutter https://github.com/aschn/cookiecutter-django-iot.git