Managing 2000+ CPEs with Celery and Django by Duane Churms

We implemented an interface between a TR069 Auto Configuration Server (ACS) and an ISP's billing system. This system ensures consistency between data on the billing system and over 2000 CPEs (Customer Premises Equipment) in the field. It prevents fraud by ensuring that credentials are only used on the devices for which they are intended. Django was used to present an API to the billing system, which can provision credentials for each CPE or deprovision a CPE. Celery is used to perform asynchronous communication with the ACS, which can take several seconds per call.

We encountered several challenges during this project, such as TR069 parameter locations being inconsistent across CPE models and firmware versions. A mapping system had to be implemented to allow all device models to be managed. This talk covers the challenges we faced and the approach we used to overcome these challenges.

Transcript

1. Managing 2000+ CPEs with Django and Celery DUANE CHURMS
2. Overview  TR069  Problem Description  Design  Challenges  Learning points
3. TR069  Technical Report 069 by the Broadband Forum  Configure Customer Premised Equipment (CPEs)  CPE WAN Management Protocol (CWMP)  Many different device types
4. Why TR069?
5. Why TR069?  Prevent vendor lock in, promote choice  Encourage competition  Sane guidelines for remote management
6. TR069 - Communications  Central Auto-Configuration Server (ACS)  SOAP/HTTP protocol  CPE initiates session with Inform request  ACS can call “webhook” on CPE
7. TR069 – Data Model  Parameters – name:value pairs  Hierarchical structure (TR 098)  e.g. InternetGatewayDevice.Time.NTPServer1  Vendors can define custom branches
8. Problem Description: Client Environment  African ISP, >2000 WiMAX CPEs  Existing FriendlyTech ACS  Very manual processes  Disconnect between Billing and real world  African power supply
9. Problem Description: Client Requirements  Allow Billing to (de)provision CPE credentials  Bulk management of CPEs:  Firmware upgrades  New frequency rollout  Improve visibility
10. Design: ACS Communication  zeep takes pain out of SOAP  ACS calls are slow  Need asynchronous execution  Celery to the rescue
11. Design: TR069 Parameters  Parameter names vary wildly  Vendor, model, firmware  ParameterMap for important parameters  All ACS calls go via ParameterMaps  Device details cached locally
12. Design: Web/API Framework  Simple API for Billing to call  Frontend for configuration, visibility  ORM for parameters  Django!
13. Design: Deployment  gunicorn + nginx  systemd service units  SaltStack  Sentry
14. Challenges: The Promise of Standards
15. Challenges: The Promise of Standards  Not just parameter name differences  Authentication  .Username = user@example.net  .Username = user .Realm = example.net  Frequency scanning table:  .FrequencyTable = 2505000,2515000,2525000  .ChannelTable.RuleXX, with .ChannelTable.InsertRule
16. Challenges: ACS Weirdness  Randomly get wrong SOAP response types  Only during bulk Celery tasks  Sentry-based investigation  ACS API clobbering request context  Stick to single Celery worker
17. Learning Points  Never trust vendors to do the right thing  Using SOAP? Reach for zeep!  Verify concurrency of upstream services  Hook up to Sentry from the start
18. Questions?