Deep dive in the Neutron upgrade story

Discover what's new in Neutron Mitaka regarding upgrade and what we plan for Newton.

Transcript

1. Deep dive into Neutron upgrade story Artur Korzeniewski Rossella Sblendido Ihar Hrachyshka
2. Agenda 1. Upgrade Team introduction 2. Neutron upgrade process overview 3. Controller plane upgrades details 4. Dataplane upgrades details 5. Future plans
3. 1.Upgrade Team introduction Formed at Tokyo summit Wiki page: https://wiki.openstack.org/wiki/Meetings/Neutron-Upgrades-Subteam Team Charter Enhances the upgrade story for Neutron Grenade CI Alembic migrations Oslo Versioned Objects RPC versioning
4. Neutron upgrade overview
5. Neutron upgrade overview 1. Get the new code 2. Upgrade database 3. Upgrade Neutron Server 4. Upgrade Network Node 5. Upgrade Compute Nodes
6. 1. Install Mitaka code of Neutron Controller node Compute node L2 agent Liberty Neutron Server Liberty Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Neutron Mitaka Neutron upgrade overview Compute node L2 agent Liberty Compute node L2 agent Liberty Database - Liberty Data
7. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand Neutron Mitaka Neutron upgrade overview Controller node Compute node L2 agent Liberty Neutron Server Liberty Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Database - Liberty Compute node L2 agent Liberty Compute node L2 agent Liberty Data
8. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server Neutron upgrade overview Controller node Compute node L2 agent Liberty Neutron Server Liberty Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Database - Liberty Compute node L2 agent Liberty Compute node L2 agent Liberty Data Offline
9. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract Neutron upgrade overview Controller node Compute node L2 agent Liberty Neutron Server Mitaka Database Data Mitaka Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Database Schema Mitaka Compute node L2 agent Liberty Compute node L2 agent Liberty Offline Controller node Compute node L2 agent Liberty Neutron Server Liberty Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Database - Mitaka Compute node L2 agent Liberty Compute node L2 agent Liberty Data Offline
10. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server Neutron upgrade overview Controller node Compute node L2 agent Liberty Neutron Server Mitaka Network node L3 Agent Liberty L2 agent Liberty Metadata Liberty DHCP Liberty Compute node L2 agent Liberty Compute node L2 agent Liberty Database - Mitaka Data
11. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Liberty Network node L3 Agent Liberty L2 agent Mitaka Metadata Liberty DHCP Liberty Compute node L2 agent Liberty Compute node L2 agent Liberty Controller node Neutron Server Mitaka Database - Mitaka Data
12. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Liberty Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Compute node L2 agent Liberty Compute node L2 agent Liberty Controller node Neutron Server Mitaka Database - Mitaka Data
13. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Mitaka Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Compute node L2 agent Liberty Compute node L2 agent Liberty Controller node Neutron Server Mitaka Database - Mitaka Data
14. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Mitaka Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Compute node L2 agent Mitaka Compute node L2 agent Liberty Controller node Neutron Server Mitaka Database - Mitaka Data
15. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Mitaka Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Compute node L2 agent Mitaka Compute node L2 agent Mitaka Controller node Neutron Server Mitaka Database - Mitaka Data
16. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview Compute node L2 agent Mitaka Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Compute node L2 agent Mitaka Compute node L2 agent Mitaka Controller node Neutron Server Mitaka Database - Mitaka Data End
17. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview - DVR case Compute node L2 agent Liberty L3 agent Liberty Metadata Liberty Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Controller node Neutron Server Mitaka Database - Mitaka Data
18. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview - DVR case Compute node L2 agent Mitaka L3 agent Liberty Metadata Liberty Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Controller node Neutron Server Mitaka Database - Mitaka Data
19. 1. Install Mitaka code of Neutron 2. Run expand phase on database schema a. neutron-db-manage upgrade --expand 3. Upgrade Neutron Server 4. Offline data migration - offline a. neutron-db-manage has_offline_migrations b. neutron-db-manage upgrade --contract 5. Start Neutron Server 6. Upgrade Network node a. L2 agent first Neutron upgrade overview - DVR case Compute node L2 agent Mitaka L3 agent Mitaka Metadata Mitaka Network node L3 Agent Mitaka L2 agent Mitaka Metadata Mitaka DHCP Mitaka Controller node Neutron Server Mitaka Database - Mitaka Data
20. Controller upgrade
21. Controller upgrade Expectations: Minimal downtime for API endpoints In HA architecture, there is no API downtime at all Controller is capable of talking to not (yet) upgraded agents
22. Controller upgrade Reality (pre Liberty): Significant downtime for API endpoints No staged upgrade for HA controllers Rolling upgrades are ‘kinda’ supported, but not really tested
23. Controller: API downtime
24. Controller upgrade: API downtime Database schema upgrade is managed by alembic Database data upgrade is managed by alembic too Most alembic scripts are unsafe to execute with controller running
25. Controller upgrade: API downtime Before Liberty: $ systemctl stop neutron-server [on all nodes] $ neutron-db-manage upgrade head $ systemctl start neutron-server [on all nodes]
26. Controller upgrade: API downtime K L L Expand Contract
27. Controller upgrade: API downtime Liberty+: $ neutron-db-manage upgrade --expand $ systemctl stop neutron-server $ neutron-db-manage upgrade --contract ← DOWNTIME!!! $ systemctl start neutron-server
28. Controller upgrade: API downtime Liberty+: $ neutron-db-manage upgrade --expand $ systemctl stop neutron-server $ neutron-db-manage upgrade --contract ← DOWNTIME!!! $ systemctl start neutron-server
29. Controller upgrade: API downtime Newton+: $ neutron-db-manage upgrade --expand $ systemctl stop neutron-server ← DOWNTIME!!! $ systemctl start neutron-server $ neutron-db-manage upgrade --contract (a lot later, assuming all data migrated)
30. Controller upgrade: issues to solve Controller cluster talking to different versions of database schema Meaning, every database access should consider that And that’s a lot of places in the code! Meaning, SQLAlchemy model is not enough as a database access façade
31. Meet oslo.versionedobjects
32. Why oslo.versionedobjects? Strict object representation for Neutron resources And it’s all versioned! RPC messages can exchange versioned objects, it’s easier to support older clients Hide database details behind the façade in a single place Data can be lazily migrated to new database schema on resource access
33. We fell in love with objects!
34. BUT "the course of true love never did run smooth" - Shakespeare
35. Neutron extensibility vs OVO strictness ❖ Neutron has plugins and extensions ➢ objects mutate according to the configuration ❖ oslo.versionedobjects require a strict definition
36. Synthetic fields Fields that don’t depend on the DB table of the object might depend on another table -> usually are type ObjectField might not be stored in the DB at all
37. Port example class Port(base.NeutronDbObject): db_model = models_v2.Port fields = { 'id': obj_fields.UUIDField(), [...] 'fixed_ips': obj_fields.ListOfObjectsField('IPAllocation, nullable=True), 'binding': obj_fields.ObjectField('PortBinding', nullable=True), addr_pair.ADDRESS_PAIRS: obj_fields.ListOfObjectsField('AllowedAddressPair', nullable=True), [...] } synthetic_fields = ['fixed_ips', 'binding', addr_pair.ADDRESS_PAIRS]
38. IPAllocation class IPAllocation(base.NeutronDbObject): db_model = models_v2.IPAllocation fields = { 'port_id': obj_fields.UUIDField(), 'subnet_id': obj_fields.UUIDField(), 'network_id': obj_fields.UUIDField(), 'ip_address': obj_fields.IPAddressField() } primary_keys = ['port_id'] foreign_keys = {'port_id': 'id'}
39. OVO introduction in Neutron A lot of refactoring involved in Neutron repositories :( Lots of patches in flight, more to come We are nevertheless optimistic to get it done in Newton Follow the progress: https://blueprints.launchpad.net/openstack/? searchtext=adopt-oslo-versioned-objects-for-db
40. Controller: HA
41. Mitaka and earlier
42. Controller upgrade: HA: pre-Newton Whole cluster running ‘old’ release neutron-server neutron-server neutron-server neutron-server
43. Controller upgrade: HA: pre-Newton To do controller upgrade, all API endpoints need to shutdown neutron-server neutron-server neutron-server neutron-server
44. Controller upgrade: HA: pre-Newton Database updated, all controllers are back neutron-server neutron-server neutron-server neutron-server
45. Newton and beyond
46. Controller upgrade: HA: Newton+ Whole cluster running ‘old’ release, database schema already upgraded to ‘new’ neutron-server neutron-server neutron-server neutron-server
47. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
48. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
49. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
50. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
51. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
52. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
53. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
54. Controller upgrade: HA: Newton+ ‘Old’ servers are ok working with the ‘new’ schema, we can stage upgrade neutron-server neutron-server neutron-server neutron-server
55. In the meantime, controllers should reply to API and RPC requests
56. Controller: internals
57. How to upgrade schema online?
58. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
59. Controller upgrade: internals neutron-server neutron-server Database schema Schema expanded N-1 N N+1 N+2
60. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
61. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
62. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
63. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
64. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
65. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
66. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
67. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
68. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
69. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
70. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
71. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
72. Controller upgrade: internals neutron-server neutron-server Database schema Now we can drop old schema! N-1 N N+1 N+2
73. Controller upgrade: internals neutron-server neutron-server Database schema N-1 N N+1 N+2
74. Upgrade: how is the dataplane affected?
75. L2 agent Runs on compute and network nodes Configures the local vbridges (br-int, br-tun) When using OVS, the OVS agent installs flows to manage network traffic
77. What’s a flow? Flow = match + action ovs-ofctl dump-flows br-int
78. The problem... To upgrade, agents need to be restarted Before Liberty the OVS agent cleared all the flows when restarted, BOOM! Solution: All the flows added get a cookie with a UUID When the agent restarts, it creates new flows and then deletes stale flows (identified by cookie with a different UUID)
79. More improvements in Mitaka VLAN/Flat provider network no connectivity loss: flows for physical bridges now get a cookie too and are not removed after restart Patch ports between br-int / br-tun and br-int / br-phys are not deleted anymore
80. And how do you validate that?
81. Upgrade: testing Grenade CI Installs latest stable release, perform Tempest smoke tests, and upgrade the code to current master and run smoke tests again New ‘partial’ (multinode) job was introduced during Mitaka Primary node running AIO: controller, compute and network node Secondary node running nova-compute and Open vSwitch agents Primary node is upgraded and the secondary node is left with older version Primary node Controller Mitaka Secondary node Compute Mitaka L2 agent Mitaka Compute Mitaka L2 agent Mitaka
82. Upgrade: testing Grenade CI Installs latest stable release, perform Tempest smoke tests, and upgrade the code to current master and run smoke tests again New ‘partial’ (multinode) job was introduced during Mitaka Primary node running AIO: controller, compute and network node Secondary node running nova-compute and Open vSwitch agents Primary node is upgraded and the secondary node is left with older version Primary node Controller Master Secondary node Compute Master L2 agent Mitaka Compute Mitaka L2 agent Master
83. Upgrade: testing Alembic scripts will get proper testing coverage in Newton no more pre-release rush to fix migrations for PostgreSQL Improvements needed: move more services to ‘old’ side of the cluster L3 agent, DHCP agent, Metadata agent Introduce DVR testing
84. Future Plans
85. Future plans 1. Newton: Complete OVO implementation 2. Newton: Improve Grenade CI 3. Newton+: Allow upgrading controllers in rolling mode 4. Newton+: Improve compatibility when running mixed node versions
86. Thanks! Questions? Operators, your feedback is needed!
87. Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2016 Intel Corporation.