API Contracts: Bringing OpenAPI and typing to OpenStack

OpenStack's APIs - of both RESTful and Python variety - have evolved organically over time. This has allowed them to grow and change as new use cases have arisen (and older ones died away), but this organic growth has resulted in a large amount of cruft, weird corner cases and bugs, and generally lots of undefined behaviour, all things commonly known as tech debt. While a lot has been done to reduce this tech debt - ranging from low-hanging fruit like improved docstrings and better unit test coverage to bigger efforts like the api-ref docs and functional API tests - we know from experience that there are a non-trivial number of latent issues just waiting to be triggered. Fortunately, the broader tech ecosystem has not stood still since OpenStack was first introduced in the early 2010s. Two technologies in particular promise to massively improve how we document and understand our APIs: OpenAPI and Python type annotations. OpenAPI promises to provide a machine-readable definition of the various APIs, while Python type annotations allow us a way to provide guarantees about type information that were not previously possible in Python. In this talk, we will explore the work that has been ongoing to integrate both technologies into projects across OpenStack, how you as a user or developer can taking advantage of them, and how you can get involved.

Transcript

1. API Contracts: Bringing OpenAPI and Typing to OpenStack Improving documentation and understanding of OpenStack’s (second) greatest asset: its APIs OpenInfra Summit Asia ‘24 @stephenfin
2. Stephen Finucane (@stephenfin) Senior Software Engineer Red Hat
4. Gophercloud
5. Background
7. OpenStackSDK (Python) ↗ Gophercloud (Go) ↗ OpenStack4j (Java) ↗ php-opencloud/openstack (PHP) ↗ …
8. Lots of dead SDKs. Why?
9. Lots of dead SDKs. Why? Developer lost interest OpenStack’s “plateau of productivity” / No longer “cool” ✨ It’s hard work™
10. There are so many services There are so many APIs 🎶 A Good Song Bad API Never Dies 🎶 …
11. Other issues OpenStackClient (and SDK) can be slooowwww… 🦀 Completeness of existing SDKs New languages, new environments Obsolete web frameworks and libraries
12. So we thought about it… 🤔
13. So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔
14. So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔 And we considered giving up (more than once)… 😫
15. So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔 And we considered giving up (more than once)… 😫 But we eventually arrived at a solution! 💪🏆🎉
17. Intro to OpenAPI in OpenStack
18. Superset of JSONSchema (draft 2020-12) Support for extensions Schemas given in JSON or YAML Support in multiple languages (validation, doc generation, …)
19. openapi: "3.1.0" info: version: "1.0.0" title: "Swagger Petstore" license: name: "MIT" servers: - url: "http://petstore.swagger.io/v1" paths: /pets: get: # ... petstore.yaml
20. # ... paths: /pets: get: summary: List all pets operationId: listPets tags: - pets parameters: - name: limit in: query description: How many items to return at one time (max 100) required: false schema: type: integer maximum: 100 format: int32 # ... petstore.yaml
21. # ... paths: /pets: get: # ... responses: '200': description: A paged array of pets headers: x-next: description: A link to the next page of responses schema: type: string content: application/json: schema: $ref: "#/components/schemas/Pets" # ... petstore.yaml
22. Q: Haven’t we tried this before?
23. Q: Haven’t we tried this before? A: Yes ↗. Many ↗ times ↗. However!
24. Superset of JSONSchema (draft 2020-12) Support for extensions Schemas given in JSON or YAML Support in multiple languages (validation, doc generation, …)
25. openstack/codegenerator ↗
27. What does it do?
28. What does it do? Generates OpenAPI schemas Generate bindings in chosen language
29. # ... paths: /v2.1/flavors/{flavor_id}/os-flavor-access: parameters: - $ref: '#/components/parameters/flavors_os_flavor_access_flavor_id' get: operationId: flavors/flavor_id/os-flavor-access:get responses: '404': description: Error '200': description: Ok content: application/json: schema: $ref: '#/components/schemas/FlavorsOs_Flavor_AccessListResponse' openapi_specs/compute/v2.96.yaml (generated file)
30. # ... components: schemas: FlavorsOs_Flavor_AccessListResponse: type: object properties: flavor_access: type: array items: type: object properties: flavor_id: type: string format: uuid tenant_id: type: string format: uuid openapi_specs/compute/v2.96.yaml (generated file with field omitted)
31. How does it do it?
32. How does it do it? Inspection of code
33. class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py
34. class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py
35. # ... # TODO(stephenfin): Remove additionalProperties in a future API version index_query = { 'type': 'object', 'properties': {}, 'additionalProperties': True, } # ... nova/api/openstack/compute/schemas/flavor_access.py
36. How does it do it? Inspection of code
37. How does it do it? Inspection of code Inspection of documentation
38. How does it do it? Inspection of code Inspection of documentation (specifically, the compiled api-ref documentation 🤮)
39. Problems api-ref documentation is not machine-readable api-ref documentation can be incomplete/outdated api-ref is reStructuredText Verification is difficult Schemas live in another project
40. We can do better!
41. Step 1: MOAR JSONSchema!
42. class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py
43. class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py
44. class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) @validation.response_body_schema(schema.index_response) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py
45. from nova.api.validation import parameter_types # ... index_response = { 'type': 'object', 'properties': { # NOTE(stephenfin): While we accept numbers as values, we always # return strings 'extra_specs': parameter_types.metadata, }, 'required': ['extra_specs'], 'additionalProperties': False, } # ... nova/api/openstack/compute/schemas/flavor_access.py
46. @wsgi.Controller.api_version(RESOURCE_LOCKS_MIN_API_VERSION) @wsgi.Controller.authorize('get_all') @validation.request_query_schema(schema.index_request_query) @validation.response_body_schema(schema.index_response_body) def index(self, req): """Returns a list of locks, transformed through view builder.""" context = req.environ['manila.context'] filters = req.params.copy() params = common.get_pagination_params(req) limit, offset = [ params.pop('limit', None), params.pop('offset', None) ] sort_key, sort_dir = common.get_sort_params(filters) for key in ('limit', 'offset'): filters.pop(key, None) # ... manila/api/v2/resource_locks.py
47. @wsgi.Controller.api_version(RESOURCE_LOCKS_MIN_API_VERSION) @wsgi.Controller.authorize('get_all') @validation.request_query_schema(schema.index_request_query) @validation.response_body_schema(schema.index_response_body) def index(self, req): """Returns a list of locks, transformed through view builder.""" context = req.environ['manila.context'] filters = req.params.copy() params = common.get_pagination_params(req) limit, offset = [ params.pop('limit', None), params.pop('offset', None) ] sort_key, sort_dir = common.get_sort_params(filters) for key in ('limit', 'offset'): filters.pop(key, None) # ... manila/api/v2/resource_locks.py
48. from manila.api.validation import helpers create_request_body = { 'type': 'object', 'properties': { 'resource_lock': { 'type': 'object', 'properties': { 'resource_id': { 'type': 'string', 'format': 'uuid', 'description': helpers.description( 'resource_lock_resource_id' ), }, }, }, }, } manila/api/schemas/resource_locks.py
49. from manila.api.validation import helpers create_request_body = { 'type': 'object', 'properties': { 'resource_lock': { 'type': 'object', 'properties': { 'resource_id': { 'type': 'string', 'format': 'uuid', 'description': helpers.description( 'resource_lock_resource_id' ), }, }, }, }, } manila/api/schemas/resource_locks.py
50. ❯ tox -e py310 functional-py310
51. Step 2: OpenStack Extensions (🔮)
52. Microversions
53. Microversions RPC-style APIs (actions)
54. Microversions RPC-style APIs (actions) Policy
55. Microversions RPC-style APIs (actions) Policy Generator hints
56. components: securitySchemes: APIGatewayAuthorizer: type: apiKey name: Authorization in: header x-amazon-apigateway-authtype: oauth2 x-amazon-apigateway-authorizer: type: token authorizerUri: ... authorizerCredentials: arn:aws:iam::account-id:role identityValidationExpression: "^x-[a-z]+" authorizerResultTtlInSeconds: 60 OpenAPI Extensions example (from swagger.io/docs/specification/openapi-extensions)
57. Step 3: Migrate tooling (🔮)
58. Tempest Documentation (os-api-ref) Clients (maybe… 🦀) …
59. And Now, Typing 🪄
60. Introduced in PEP 484 ↗ Expanded in multiple subsequent PEPs (526, 544, 560, …↗) And still being expanded…
61. def greet(name): print(f'Hello, {name}') greet.py
62. def greet(name: str) -> None: print(f'Hello, {name}') greet.py
63. class Formatter: @classmethod def deserialize(cls, value): """Return a formatted object representing the value""" raise NotImplementedError class BoolStr(Formatter): @classmethod def deserialize(cls, value): """Convert a boolean string to a boolean""" expr = str(value).lower() if "true" == expr: return True elif "false" == expr: return False else: raise ValueError("Invalid value: %s" % value) openstack/format.py (from openstack/openstacksdk)
64. class Formatter(ty.Generic[_T]): @classmethod def deserialize(cls, value: str) -> _T: """Return a formatted object representing the value""" raise NotImplementedError class BoolStr(Formatter[bool]): @classmethod def deserialize(cls, value: str) -> bool: """Convert a boolean string to a boolean""" expr = str(value).lower() if "true" == expr: return True elif "false" == expr: return False else: raise ValueError("Invalid value: %s" % value) openstack/format.py (from openstack/openstacksdk)
65. Q: What does this have to do with OpenAPI?
66. Q: What does this have to do with OpenAPI? A: Nothing, really. But it’s helpful in many ways…
68. from openstack.format import BoolStr value = 123 BoolStr.deserialize(value) test.py
70. What else? Remove typing-related tests Generate actually helpful Python API docs Better understanding of legacy code Cross-validation with OpenAPI docs …
71. Where? OpenStackSDK (4/10) Nova (2/10) Cinder (2/10?) keystoneauth1 (0/10, but soon 🔮) …
72. And So, To Conclude…
73. OpenAPI is coming to OpenStack
74. OpenAPI is coming to OpenStack Typing is coming to OpenStack
75. OpenAPI is coming to OpenStack Typing is coming to OpenStack black ruff-format is coming to OpenStack (?) 🙃
76. Want to get involved? #openstack-sdk on OFTC IRC openstack-discuss@lists.openstack.org Catch us in person
77. API Contracts: Bringing OpenAPI and Typing to OpenStack Improving documentation and understanding of OpenStack’s (second) greatest asset: its APIs OpenInfra Summit Asia ‘24 @stephenfin
78. Credits Cover photo by Giammarco Boscaro on Unsplash Source code from openstacksdk, Nova and Manila projects (Apache 2.0)