PyConZA 2012: "Building RESTful, service-oriented architectures with Twisted" by Bryn Divey

General overview of building service-oriented systems using REST and Twisted. Motivations for doing so, design considerations, issues encountered, possible mitigations. Will try and pull it back to Twisted as far as possible.

Transcript

1. Building RESTful, Service- Oriented Architectures with Twisted Bryn Divey, Senior Developer Nimbula bryn@nimbula.com Friday, October 12, 2012
2. Introduction Friday, October 12, 2012
3. Me • Bryn Divey • Senior dev. at Nimbula • Worked in Python professionally for about 10 years • Worked with Twisted professionally for the last 4 years Friday, October 12, 2012
4. Nimbula • Cloud Operating System • Distributed system (no SPOFs) running at scale (100s of machines currently, aiming for 10s of thousands) • Creates a private cloud fabric and API over datacenters for virtualization • Written in Python/Twisted Friday, October 12, 2012
5. Nimbula • Scale • ~20 primary services which interact, each having two to fifteen instances depending on the size of the site • ~5 node level services, running on every machine in the size Friday, October 12, 2012
6. Nimbula • Large amounts of internal communication • Metrics • Auditing • Security is vital Friday, October 12, 2012
7. Concepts And Pieces Friday, October 12, 2012
8. Service Oriented Architecture Friday, October 12, 2012
9. SOA • Garter - 1996 (possibly) • Architecture style to build large resilient systems, often by large numbers of developers Friday, October 12, 2012
10. SOA • Isolate responsibilities into separate services • Establish explicit boundaries of communication • Loose coupling • Statelessness • Discoverability Friday, October 12, 2012
11. Representational State Transfer (REST) Friday, October 12, 2012
12. REST • Roy Fielding - 2000 • Client/server architecture for the transfer and manipulation of resources • What the web is based on Friday, October 12, 2012
13. REST • Client-server architecture - simplifies implementation of both • Navigable - transition between resources using links (hypertext) • Stateless - client state not persisted between requests • Cacheable - conditions defined by server Friday, October 12, 2012
14. REST • Layered - intermediary servers may modify requests on the way, or provide caching or load balancing services • Uniform interface - defined methods (GET, PUT, etc.), standard response codes (2xx, 4xx, 5xx), well-known content-types Friday, October 12, 2012
15. REST • Clients manipulate uniquely identified resources on servers using requests comprising methods (POST) and representations of the resource (JSON representation of a server object that may be stored in a different form) Friday, October 12, 2012
16. Twisted Friday, October 12, 2012
17. Twisted • Guys with names like “glyph”, “itamar” and “exarkun” - 2001 • Event-driven networking framework Friday, October 12, 2012
18. Twisted - Event Driven • Reactor which watches underlying file handles efficiently (uses epoll on Linux) • Calls back into application code on change (eg. data received on a socket) Friday, October 12, 2012
19. Twisted - Event Driven • Developers write callbacks to react to the result of IO operations • or the result of threaded operations • or future scheduled calls Friday, October 12, 2012
20. Twisted - Event Driven • Other application code is scheduled to run while waiting on async IO • Allows network heavy applications to “simultaneously” service multiple requests in a single thread, scheduling action on IO Friday, October 12, 2012
21. Twisted • Use of callbacks for future processing • Separation of protocols and transports • Huge library of protocols and utilities Friday, October 12, 2012
22. Motivations For These Choices Friday, October 12, 2012
23. Why SOA? Friday, October 12, 2012
24. Why SOA ? • Modularity • New systems/features are contained within their own service • Enforces good interfaces • Having defined interfaces between sections of code is good practice; using services enforces it Friday, October 12, 2012
25. Why SOA ? • Redundancy/request scaling • With services running across hundreds of machines failure is expected - having multiple instances of services makes it non-disruptive • Freedom wrt to implementation • No lock in to language or framework - allows for eg. using compiled languages for performance critical sections Friday, October 12, 2012
26. Why REST? Friday, October 12, 2012
27. Why REST? • Simple model <-> API mapping • Designing APIs is hard. Designing models and choosing which HTTP verbs apply to them is a little easier • Also, forces developers to simplify their APIs (no EmailUsersOfFeatureX calls - must create or manipulate a model) Friday, October 12, 2012
28. Why REST? • Ability to overlay industry standard HTTP stuff • caches, proxies, etc. • Easy to apply middleware • eg. authentication • Seemed like a good idea at the time • were designing for WANs not LANs • REST was the coolest thing in 2008 Friday, October 12, 2012
29. Why Twisted? Friday, October 12, 2012
30. Twisted • Guys with names like “glyph”, “itamar” and “exarkun” • Battle-proven framework • No threading complexities • Initial developers has familiarity • Thought we could isolated unfamiliar devs using threadPools and inlineCallbacks Friday, October 12, 2012
31. Approach Friday, October 12, 2012
32. Approach • Split up services by concern • ‘user’ service handled all user, group, authentication type calls • ‘instance’ service places, runs, and destroys virtual machines • ‘storage’ service manages storage servers and attaches virtual block storage Friday, October 12, 2012
33. Approach • Create a product wide library of models that represent all aspects of the system • as the API is a rather CRUDdy version of REST, most methods on these object match up to HTTP verbs: get, list, update, save, delete • this makes models easy to write with metaclasses consuming their field definitions Friday, October 12, 2012
34. Approach • Abstract away protocol and codec • Generic controller per model • Operations take a model and metadata, return a response or raise an exception • Details of HTTP transport hidden from controller • Allows for HTTP to be swapped out for eg. AQMP invisibly Friday, October 12, 2012
35. Approach • Abstract away protocol and codec • Allows different access patterns depending on context • user.get() in a separate controller will cause an HTTP call • user.get() in the user service will query the underlying database • calls are identical, sources change Friday, October 12, 2012
36. Approach • Services handle a related subset of models • user service handles the User, Group, AuthenticationToken models Friday, October 12, 2012
37. Approach • Routing handled by DNS name and URI components • start with a well-known endpoint: https:// api.cloud Friday, October 12, 2012
38. Approach • Front-end HTTP proxy routes to the appropriate service • models.Instance.list() -> front-end • GET http://api.cloud/compute/instance -> compute service • GET http://compute.cloud:5001/compute/instance • nginx acts as a service bus - routes requests with ‘compute’ base URI to the compute service DNS Friday, October 12, 2012
39. Approach • Independent scalability • when services are under pressure, start a new instance on an arbitrary node on the site • new instances load-balanced to using DNS round-robin Friday, October 12, 2012
40. Example Friday, October 12, 2012
41. Examples • Creating a new virtual machine: does the user have access to the machine image? Friday, October 12, 2012
42. try: yield machineimage = models.MachineImage.get( 'c323c3ee-cc86-47c0-8518-59a7ee02a06a') except APIUnauthorized: raise Invalid("User is not authorized to use" " machine image 'c323c3ee-cc86-47c0-8518-59a7ee02a06a'") GET https://api.cloud/images/machineimage/c323c3ee- cc86-47c0-8518-59a7ee02a06a proxies to GET https://image:5005/images/machineimage/ c323c3ee-cc86-47c0-8518-59a7ee02a06a returns 200 OK - with machine image details on success returns 401 Unauthorized - if the user doesn't have access Examples Friday, October 12, 2012
43. Approach • And with Twisted it’s trivial to parallelize calls to multiple external blocking services: get machine images and storage volumes, and check if there are conflicting virtual machines (an operation on the database local to the service) Friday, October 12, 2012
44. Examples @inlineCallbacks def process_external_resources(instance): # each returns a list of deferreds machine_images = get_machine_images_for_instance (instance) storage_volumes = get_storage_volumes_for_instance (instance) related_instances = get_related_instances(instance) # all deferreds are processed in parallel, so this call (hypothetically) # takes only as long as the slowest one results = yield gatherResults(machine_images + storage_volumes + related_instances) Friday, October 12, 2012
45. Approach • This function results in a number of HTTP calls to both the image and storage services, and local DB queries • All call signatures to fetch objects are the same • All IO operations happen in parallel, and the thread of execution does something else until they finish Friday, October 12, 2012
46. Issues Friday, October 12, 2012
47. SOA • Sometimes difficult to decide which service has responsibility over an objects • Some actions require a significant amount of interservice communication Friday, October 12, 2012
48. REST • Some actions are difficult to model within a resource framework • Leads to the creation of “action” models Friday, October 12, 2012
49. REST • Multi-object calls become really expensive • Doing 4000 auth requests, fetching 1000 machine images, creating 1000 quota objects etc. to launch 1000 instances • Each requires an HTTP request, requiring an SSL encrypted connection, and these have to be limited. Each connection is blocked until the current request completes. Friday, October 12, 2012
50. REST • Multi-object calls become really expensive • Actually become a major problem Friday, October 12, 2012
51. Twisted • Onboarding engineers • Our problem area means we hire a lot of low- level devs • Moving across to the idioms and libraries of Python, from C, can sometimes be difficult • Adding learning Twisted on top makes is even more so Friday, October 12, 2012
52. Twisted • Explicit deferred handling • Thought inlineCallbacks would help • Forgetting a yield, or the decorator, causes horrible problems • Currently we have 6415 yields in the codebase - 6413 of them in inlineCallbacks functions • Very difficult to track down - needs static analysis Friday, October 12, 2012
53. Conclusions Friday, October 12, 2012
54. Conclusions • REST is great, but perhaps more so for internal APIs • Personally I’d prefer something that topic routes, and does multiplexing on connections (AMQP!) Friday, October 12, 2012
55. Conclusions • Related, be very aware of what’s happening in the low-level routing • We get connection pools locking up, requests timing out, etc. • Monitor these things and throw loud alerts Friday, October 12, 2012
56. Conclusions • Twisted • Awesome framework • But if you’re not going to use half of the protocols, it may be easier to use something like gevent instead Friday, October 12, 2012