API Contracts: Bringing OpenAPI and typing to OpenStack

Slide 1

Slide 1 text

API Contracts: Bringing OpenAPI and Typing to OpenStack Improving documentation and understanding of OpenStack’s (second) greatest asset: its APIs OpenInfra Summit Asia ‘24 @stephenﬁn

Slide 2

Slide 2 text

Stephen Finucane (@stephenﬁn) Senior Software Engineer Red Hat

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Gophercloud

Slide 5

Slide 5 text

Background

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

OpenStackSDK (Python) ↗ Gophercloud (Go) ↗ OpenStack4j (Java) ↗ php-opencloud/openstack (PHP) ↗ …

Slide 8

Slide 8 text

Lots of dead SDKs. Why?

Slide 9

Slide 9 text

Lots of dead SDKs. Why? Developer lost interest OpenStack’s “plateau of productivity” / No longer “cool” ✨ It’s hard work™

Slide 10

Slide 10 text

There are so many services There are so many APIs 🎶 A Good Song Bad API Never Dies 🎶 …

Slide 11

Slide 11 text

Other issues OpenStackClient (and SDK) can be slooowwww… 🦀 Completeness of existing SDKs New languages, new environments Obsolete web frameworks and libraries

Slide 12

Slide 12 text

So we thought about it… 🤔

Slide 13

Slide 13 text

So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔

Slide 14

Slide 14 text

So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔 And we considered giving up (more than once)… 😫

Slide 15

Slide 15 text

So we thought about it… 🤔 And we thought about it some more… 🤔🤔🤔 And we considered giving up (more than once)… 😫 But we eventually arrived at a solution! 💪🏆🎉

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Intro to OpenAPI in OpenStack

Slide 18

Slide 18 text

Superset of JSONSchema (draft 2020-12) Support for extensions Schemas given in JSON or YAML Support in multiple languages (validation, doc generation, …)

Slide 19

Slide 19 text

openapi: "3.1.0" info: version: "1.0.0" title: "Swagger Petstore" license: name: "MIT" servers: - url: "http://petstore.swagger.io/v1" paths: /pets: get: # ... petstore.yaml

Slide 20

Slide 20 text

# ... paths: /pets: get: summary: List all pets operationId: listPets tags: - pets parameters: - name: limit in: query description: How many items to return at one time (max 100) required: false schema: type: integer maximum: 100 format: int32 # ... petstore.yaml

Slide 21

Slide 21 text

# ... paths: /pets: get: # ... responses: '200': description: A paged array of pets headers: x-next: description: A link to the next page of responses schema: type: string content: application/json: schema: $ref: "#/components/schemas/Pets" # ... petstore.yaml

Slide 22

Slide 22 text

Q: Haven’t we tried this before?

Slide 23

Slide 23 text

Q: Haven’t we tried this before? A: Yes ↗. Many ↗ times ↗. However!

Slide 24

Slide 24 text

Superset of JSONSchema (draft 2020-12) Support for extensions Schemas given in JSON or YAML Support in multiple languages (validation, doc generation, …)

Slide 25

Slide 25 text

openstack/codegenerator ↗

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

What does it do?

Slide 28

Slide 28 text

What does it do? Generates OpenAPI schemas Generate bindings in chosen language

Slide 29

Slide 29 text

# ... paths: /v2.1/flavors/{flavor_id}/os-flavor-access: parameters: - $ref: '#/components/parameters/flavors_os_flavor_access_flavor_id' get: operationId: flavors/flavor_id/os-flavor-access:get responses: '404': description: Error '200': description: Ok content: application/json: schema: $ref: '#/components/schemas/FlavorsOs_Flavor_AccessListResponse' openapi_specs/compute/v2.96.yaml (generated ﬁle)

Slide 30

Slide 30 text

# ... components: schemas: FlavorsOs_Flavor_AccessListResponse: type: object properties: flavor_access: type: array items: type: object properties: flavor_id: type: string format: uuid tenant_id: type: string format: uuid openapi_specs/compute/v2.96.yaml (generated ﬁle with ﬁeld omitted)

Slide 31

Slide 31 text

How does it do it?

Slide 32

Slide 32 text

How does it do it? Inspection of code

Slide 33

Slide 33 text

class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py

Slide 34

Slide 34 text

Slide 35

Slide 35 text

# ... # TODO(stephenfin): Remove additionalProperties in a future API version index_query = { 'type': 'object', 'properties': {}, 'additionalProperties': True, } # ... nova/api/openstack/compute/schemas/flavor_access.py

Slide 36

Slide 36 text

How does it do it? Inspection of code

Slide 37

Slide 37 text

How does it do it? Inspection of code Inspection of documentation

Slide 38

Slide 38 text

How does it do it? Inspection of code Inspection of documentation (speciﬁcally, the compiled api-ref documentation 🤮)

Slide 39

Slide 39 text

Problems api-ref documentation is not machine-readable api-ref documentation can be incomplete/outdated api-ref is reStructuredText Veriﬁcation is difﬁcult Schemas live in another project

Slide 40

Slide 40 text

We can do better!

Slide 41

Slide 41 text

Step 1: MOAR JSONSchema!

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

class FlavorAccessController(wsgi.Controller): """The flavor access API controller for the OpenStack API.""" @wsgi.expected_errors(404) @validation.query_schema(schema.index_query) @validation.response_body_schema(schema.index_response) def index(self, req, flavor_id): context = req.environ['nova.context'] context.can(fa_policies.BASE_POLICY_NAME) flavor = common.get_flavor(context, flavor_id) # public flavor to all projects if flavor.is_public: explanation = _("...") raise webob.exc.HTTPNotFound(explanation=explanation) # private flavor to listed projects only return _marshall_flavor_access(flavor) nova/api/openstack/compute/flavor_access.py

Slide 45

Slide 45 text

from nova.api.validation import parameter_types # ... index_response = { 'type': 'object', 'properties': { # NOTE(stephenfin): While we accept numbers as values, we always # return strings 'extra_specs': parameter_types.metadata, }, 'required': ['extra_specs'], 'additionalProperties': False, } # ... nova/api/openstack/compute/schemas/flavor_access.py

Slide 46

Slide 46 text

@wsgi.Controller.api_version(RESOURCE_LOCKS_MIN_API_VERSION) @wsgi.Controller.authorize('get_all') @validation.request_query_schema(schema.index_request_query) @validation.response_body_schema(schema.index_response_body) def index(self, req): """Returns a list of locks, transformed through view builder.""" context = req.environ['manila.context'] filters = req.params.copy() params = common.get_pagination_params(req) limit, offset = [ params.pop('limit', None), params.pop('offset', None) ] sort_key, sort_dir = common.get_sort_params(filters) for key in ('limit', 'offset'): filters.pop(key, None) # ... manila/api/v2/resource_locks.py

Slide 47

Slide 47 text

Slide 48

Slide 48 text

from manila.api.validation import helpers create_request_body = { 'type': 'object', 'properties': { 'resource_lock': { 'type': 'object', 'properties': { 'resource_id': { 'type': 'string', 'format': 'uuid', 'description': helpers.description( 'resource_lock_resource_id' ), }, }, }, }, } manila/api/schemas/resource_locks.py

Slide 49

Slide 49 text

Slide 50

Slide 50 text

❯ tox -e py310 functional-py310

Slide 51

Slide 51 text

Step 2: OpenStack Extensions (🔮)

Slide 52

Slide 52 text

Microversions

Slide 53

Slide 53 text

Microversions RPC-style APIs (actions)

Slide 54

Slide 54 text

Microversions RPC-style APIs (actions) Policy

Slide 55

Slide 55 text

Microversions RPC-style APIs (actions) Policy Generator hints

Slide 56

Slide 56 text

components: securitySchemes: APIGatewayAuthorizer: type: apiKey name: Authorization in: header x-amazon-apigateway-authtype: oauth2 x-amazon-apigateway-authorizer: type: token authorizerUri: ... authorizerCredentials: arn:aws:iam::account-id:role identityValidationExpression: "^x-[a-z]+" authorizerResultTtlInSeconds: 60 OpenAPI Extensions example (from swagger.io/docs/speciﬁcation/openapi-extensions)

Slide 57

Slide 57 text

Step 3: Migrate tooling (🔮)

Slide 58

Slide 58 text

Tempest Documentation (os-api-ref) Clients (maybe… 🦀) …

Slide 59

Slide 59 text

And Now, Typing 🪄

Slide 60

Slide 60 text

Introduced in PEP 484 ↗ Expanded in multiple subsequent PEPs (526, 544, 560, …↗) And still being expanded…

Slide 61

Slide 61 text

def greet(name): print(f'Hello, {name}') greet.py

Slide 62

Slide 62 text

def greet(name: str) -> None: print(f'Hello, {name}') greet.py

Slide 63

Slide 63 text

class Formatter: @classmethod def deserialize(cls, value): """Return a formatted object representing the value""" raise NotImplementedError class BoolStr(Formatter): @classmethod def deserialize(cls, value): """Convert a boolean string to a boolean""" expr = str(value).lower() if "true" == expr: return True elif "false" == expr: return False else: raise ValueError("Invalid value: %s" % value) openstack/format.py (from openstack/openstacksdk)

Slide 64

Slide 64 text

class Formatter(ty.Generic[_T]): @classmethod def deserialize(cls, value: str) -> _T: """Return a formatted object representing the value""" raise NotImplementedError class BoolStr(Formatter[bool]): @classmethod def deserialize(cls, value: str) -> bool: """Convert a boolean string to a boolean""" expr = str(value).lower() if "true" == expr: return True elif "false" == expr: return False else: raise ValueError("Invalid value: %s" % value) openstack/format.py (from openstack/openstacksdk)

Slide 65

Slide 65 text

Q: What does this have to do with OpenAPI?

Slide 66

Slide 66 text

Q: What does this have to do with OpenAPI? A: Nothing, really. But it’s helpful in many ways…

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

from openstack.format import BoolStr value = 123 BoolStr.deserialize(value) test.py

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

What else? Remove typing-related tests Generate actually helpful Python API docs Better understanding of legacy code Cross-validation with OpenAPI docs …

Slide 71

Slide 71 text

Where? OpenStackSDK (4/10) Nova (2/10) Cinder (2/10?) keystoneauth1 (0/10, but soon 🔮) …

Slide 72

Slide 72 text

And So, To Conclude…

Slide 73

Slide 73 text

OpenAPI is coming to OpenStack

Slide 74

Slide 74 text

OpenAPI is coming to OpenStack Typing is coming to OpenStack

Slide 75

Slide 75 text

OpenAPI is coming to OpenStack Typing is coming to OpenStack black ruff-format is coming to OpenStack (?) 🙃

Slide 76

Slide 76 text

Want to get involved? #openstack-sdk on OFTC IRC [email protected] Catch us in person

Slide 77

Slide 77 text

API Contracts: Bringing OpenAPI and Typing to OpenStack Improving documentation and understanding of OpenStack’s (second) greatest asset: its APIs OpenInfra Summit Asia ‘24 @stephenﬁn

Slide 78

Slide 78 text

Credits Cover photo by Giammarco Boscaro on Unsplash Source code from openstacksdk, Nova and Manila projects (Apache 2.0)