Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Managing 2000+ CPEs with Celery and Django by Duane Churms

7b0645f018c0bddc8ce3900ccc3ba70c?s=47 Pycon ZA
October 06, 2017

Managing 2000+ CPEs with Celery and Django by Duane Churms

We implemented an interface between a TR069 Auto Configuration Server (ACS) and an ISP's billing system. This system ensures consistency between data on the billing system and over 2000 CPEs (Customer Premises Equipment) in the field. It prevents fraud by ensuring that credentials are only used on the devices for which they are intended. Django was used to present an API to the billing system, which can provision credentials for each CPE or deprovision a CPE. Celery is used to perform asynchronous communication with the ACS, which can take several seconds per call.

We encountered several challenges during this project, such as TR069 parameter locations being inconsistent across CPE models and firmware versions. A mapping system had to be implemented to allow all device models to be managed. This talk covers the challenges we faced and the approach we used to overcome these challenges.

7b0645f018c0bddc8ce3900ccc3ba70c?s=128

Pycon ZA

October 06, 2017
Tweet

Transcript

  1. Managing 2000+ CPEs with Django and Celery DUANE CHURMS

  2. Overview  TR069  Problem Description  Design  Challenges

     Learning points
  3. TR069  Technical Report 069 by the Broadband Forum 

    Configure Customer Premised Equipment (CPEs)  CPE WAN Management Protocol (CWMP)  Many different device types
  4. Why TR069?

  5. Why TR069?  Prevent vendor lock in, promote choice 

    Encourage competition  Sane guidelines for remote management
  6. TR069 - Communications  Central Auto-Configuration Server (ACS)  SOAP/HTTP

    protocol  CPE initiates session with Inform request  ACS can call “webhook” on CPE
  7. TR069 – Data Model  Parameters – name:value pairs 

    Hierarchical structure (TR 098)  e.g. InternetGatewayDevice.Time.NTPServer1  Vendors can define custom branches
  8. Problem Description: Client Environment  African ISP, >2000 WiMAX CPEs

     Existing FriendlyTech ACS  Very manual processes  Disconnect between Billing and real world  African power supply
  9. Problem Description: Client Requirements  Allow Billing to (de)provision CPE

    credentials  Bulk management of CPEs:  Firmware upgrades  New frequency rollout  Improve visibility
  10. Design: ACS Communication  zeep takes pain out of SOAP

     ACS calls are slow  Need asynchronous execution  Celery to the rescue
  11. Design: TR069 Parameters  Parameter names vary wildly  Vendor,

    model, firmware  ParameterMap for important parameters  All ACS calls go via ParameterMaps  Device details cached locally
  12. Design: Web/API Framework  Simple API for Billing to call

     Frontend for configuration, visibility  ORM for parameters  Django!
  13. Design: Deployment  gunicorn + nginx  systemd service units

     SaltStack  Sentry
  14. Challenges: The Promise of Standards

  15. Challenges: The Promise of Standards  Not just parameter name

    differences  Authentication  .Username = user@example.net  .Username = user .Realm = example.net  Frequency scanning table:  .FrequencyTable = 2505000,2515000,2525000  .ChannelTable.RuleXX, with .ChannelTable.InsertRule
  16. Challenges: ACS Weirdness  Randomly get wrong SOAP response types

     Only during bulk Celery tasks  Sentry-based investigation  ACS API clobbering request context  Stick to single Celery worker
  17. Learning Points  Never trust vendors to do the right

    thing  Using SOAP? Reach for zeep!  Verify concurrency of upstream services  Hook up to Sentry from the start
  18. Questions?