Building the OrionVM API with Grape

A presentation I did at the Sydney Ruby meetup on 14th August, 2012 about how I built the OrionVM API

Transcript

1. Building REST APIs with Grape. Lessons learnt from building the API Wednesday, 15 August 12
2. Ivan Vanderbyl Founder, CrashLog.io / TestPilot CI Interaction Designer, OrionVM @IvanVanderbyl github.com/ivanvanderbyl ivan.ly Wednesday, 15 August 12 The last time I did a presentation at rorosyd was two and half years ago, it went terribly.
3. {grape} github.com/intridea/grape A full featured Rack mountable API engine a.k.a Sinatra for APIs Wednesday, 15 August 12 So I'm back to talk about grape
4. Requirements ‣ Object representation ‣ CRUD operations ‣ Versioning ‣ Authentication ‣ State transitions ‣ Asynchronous state representation Wednesday, 15 August 12 OrionVM gave me the simple brief of: Keep it simple, and make it fast. When we set out to architect the API we had some very basic goals in mind, but they quickly grew and I'll share some of most valuable points: - Every bit of customer interaction with the Orion platform had to be possible through the API - This included not only the typical CRUD operations I'm sure you're all aware off that you get out of the box with Rails, but also the state transitions associated with physical resources — it is stopped, running, shutting down? - We made a concerted effort to version the API from day one, to ensure we could change the schema without affecting our, now quite notable, clients. - While I was building this we had the third largest telco building a Node.js client to connect with their new cloud platform. - One of the mistakes we made with the first platform's API was using account login details to authenticate the API — this nearly never represents the ideal real world use case — the API is not being accessed by a user, rather by a machine, these machines should have their own authentication with your service through API tokens. - Representing the state of transient resources like Instances, Disks and Network interfaces is tricky with a stateless API, we solved this in two ways, one, by using a caching layer to allow multiple repeated requests to resource state, and two by implementing a callback API using web hooks to allow external services to be told about state changes, instead of asking.
5. Be Your Own Client Wednesday, 15 August 12 The next big lesson was don't assume you made the right design decision because it looked good at the time, actually use your own API as you build it, I learnt this one really early on in the piece and decided to build the web frontend which would later become our management console at the same time as building the API — for this I chose Ember.js and designed it to run completely standalone to the API server, while utilising all the features of the API - The other 'client' we used was good old curl, if your queries are well structured you form a certain closeness with the use of curl, you can't get much more bare bones than that.
6. Pro Tip gem install json $ curl localhost:3000/api/instances | prettify_json.rb Wednesday, 15 August 12 The next big lesson was don't assume you made the right design decision because it looked good at the time, actually use your own API as you build it, I learnt this one really early on in the piece and decided to build the web frontend which would later become our management console at the same time as building the API — for this I chose Ember.js and designed it to run completely standalone to the API server, while utilising all the features of the API - The other 'client' we used was good old curl, if your queries are well structured you form a certain closeness with the use of curl, you can't get much more bare bones than that.
7. Document Everything developer.orionvm.com.au AS YOU GO Wednesday, 15 August 12 Document everything, every endpoint, every header, every supported parameter, every response value, and better yet make your API documentation interactive, we're yet to do this but it is on the way. Writing documentation for APIs is painful, it is even more painful when you have to do it afterwards. We found a nice balance, define a simple markdown format that represents what we want, then get into the rhythm of documenting endpoints as you write them. Have a one command deployment solution to deploying documentation, we use a simple git hook on a remote server to compile the site from markdown using nanoc then point the web server to the new source.
8. Test Driven API Development Tools: ‣ RSpec ‣ Rack::Test ‣ json_spec Wednesday, 15 August 12 If your method for testing the API you're building involved reloading curl or a web browser, you are doing things seriously wrong. With a few simple tools you can test the JSON responses from you API with very little code, and I'll show you an example at the end.
9. Building the API Wednesday, 15 August 12
10. GET /api/instances { "instances": [ { "id": "efb43283-73ce-407e-8aa5-9df645d36947", "memory": 32768, "name": "autumn-river-75.orionvm.net.au", "state": "stopped", "created_at": "2012-07-22T10:04:11Z", "updated_at": "2012-07-24T15:01:49Z", "bootable": true, "volumes": [ { "id": "2ba69b79-f088-4040-9a6c-4431303e2ebd", "name": "Ubuntu 1204 Final", "target": 0, "attach_type": "RW", "state": "provisioned", "instance_id": "efb43283-73ce-407e-8aa5-9df645d36947", "mirror_count": 2, "size": 50.0, "created_at": "2012-07-22T10:07:52Z", "updated_at": "2012-07-22T12:35:58Z" } ] } ] } From zero to response Wednesday, 15 August 12 So enough about the design, lets see some code.
11. # app/api/instances_api.rb class InstancesAPI < Grape::API ... end # config/routes.rb mount InstancesAPI => '/api' Wednesday, 15 August 12 At the core of it, every Grape app is just rack middleware, you can subclass it, initialise it, and mount it within rack, or an existing Rails app with ease, and it just works.
12. class InstancesAPI < Grape::API version 'v1', :using => :param, :parameter => "v" end http://localhost:9292/instances?v=v1 Parameter based versioning Wednesday, 15 August 12 Grape comes with three built in versioning mechanisms, you can use parameter based version, that is, when you specify the version as http parameter. The second is path based, which is common with services like github, foursquare and twitter, but considered by many to be a messy and limiting approach, which can in some cased require extra work to use certain client libraries, especially if the library is out of date and you want to update it. We chose to use header based versioning, this is by far the simplest and most flexible, if you don't specify a version it assumes you want the latest version, and if you request a specific version it gives it to you. It also means we can change the response schema without changing the endpoints, which again is very useful if we decide we want to add an attribute, or change something drastically to meet our needs without affecting existing users.
13. class InstancesAPI < Grape::API version 'v1', :using => :path end http://localhost:9292/v1/instances Path based versioning Wednesday, 15 August 12 Grape comes with three built in versioning mechanisms, you can use parameter based version, that is, when you specify the version as http parameter. The second is path based, which is common with services like github, foursquare and twitter, but considered by many to be a messy and limiting approach, which can in some cased require extra work to use certain client libraries, especially if the library is out of date and you want to update it. We chose to use header based versioning, this is by far the simplest and most flexible, if you don't specify a version it assumes you want the latest version, and if you request a specific version it gives it to you. It also means we can change the response schema without changing the endpoints, which again is very useful if we decide we want to add an attribute, or change something drastically to meet our needs without affecting existing users.
14. class InstancesAPI < Grape::API version 'v1', :using => :header, :vendor => 'orion-vm' end "Accept=application/vnd.orion-­‐vm-­‐v1+json" Header based versioning Wednesday, 15 August 12 Grape comes with three built in versioning mechanisms, you can use parameter based version, that is, when you specify the version as http parameter. The second is path based, which is common with services like github, foursquare and twitter, but considered by many to be a messy and limiting approach, which can in some cased require extra work to use certain client libraries, especially if the library is out of date and you want to update it. We chose to use header based versioning, this is by far the simplest and most flexible, if you don't specify a version it assumes you want the latest version, and if you request a specific version it gives it to you. It also means we can change the response schema without changing the endpoints, which again is very useful if we decide we want to add an attribute, or change something drastically to meet our needs without affecting existing users.
15. class InstancesAPI < Grape::API resource '/instances' do desc "List all instances" get do current_account.instances.all end desc "Update a resource" put "/:id" do instance.update_attributes(params[:instance]) end end end Defining Endpoints Wednesday, 15 August 12 This will automatically return a JSON representation of the return value of the proc, so in this case we get an array of instances at the index method, and a singular object representing the updated instance
16. class InstancesAPI < Grape::API resource '/instances' do desc "Update a resource" params do requires :instance, :type => Hash, :desc => "The Instance" end put "/:id" do instance.update_attributes(params[:instance]) end end end Validating params Wednesday, 15 August 12 Grape also provides built in validations for attributes, and will automatically send back the correct response when the request arguments are invalid, including a 400 bad request status code, and a message formatted in JSON
17. describe InstancesAPI do include Rack::Test::Methods describe 'GET /api/instances' do it 'returns an array of instances' do get "/api/instances" last_response.should be_ok last_response.should have_json_path('instances/0/hostname') end end end Testing Endpoints Wednesday, 15 August 12 Let me talk you through what is happening here, I'm sure you all the the rspec syntax so I won't delve into that, first I've included Rack Test methods, this is a simple gem for testing http requests made within rack apps, and in this case grape.
18. describe InstancesAPI do include Rack::Test::Methods describe 'GET /api/instances' do it 'returns an array of instances' do get "/api/instances" last_response.should be_ok last_response.should have_json_path('instances/0/hostname') end end end Testing Endpoints Wednesday, 15 August 12 The next part is simply describing, in a somewhat verbose way, what the endpoint we should do, this will help anyone else that works on this codebase know exactly what this example should do.
19. describe InstancesAPI do include Rack::Test::Methods describe 'GET /api/instances' do it 'returns an array of instances' do get "/api/instances" last_response.should be_ok last_response.should have_json_path('instances/0/hostname') end end end Testing Endpoints Wednesday, 15 August 12 Rack test gives as similar methods to RSpec's controller test integration, so we can simply make a get request at this endpoint and it will capture the response for us.
20. describe InstancesAPI do include Rack::Test::Methods describe 'GET /api/instances' do it 'returns an array of instances' do get "/api/instances" last_response.should be_ok last_response.should have_json_path('instances/0/hostname') end end end Testing Endpoints Wednesday, 15 August 12 And lastly, we check that the response had a 200 OK response, and that the returned JSON contains the path instances, which is an array, and the first item has the key hostname.
21. {"id":1} Structuring responses [{"id":1}, {"id":2}, {"id":3}] Single object Collection of objects What sort of object is this? Wednesday, 15 August 12 You might think JSON is JSON, so allow me to point out some fine nuances of JSON responses. Let's take the single object approach, we have an object, pretty simple, and we have a collection of objects, pretty simple again. Now what is this object?
22. {"instance": {"id":1}} Structuring responses {"instances": [{"id":1}, {"id":2}, {"id":3}]} Single Instance Collection of Instances Always include a root element Wednesday, 15 August 12 Always include a root element, root elements identify to your machines what sort of object they are consuming
23. {"instance": {"id":1}} Structuring responses {"instances": [{"id":1}, {"id":2}, {"id":3}]} Single Instance Collection of Instances No Object root element in collections {"instances": [{"instance"{"id":1}}} Collection of Instances Don't do this Wednesday, 15 August 12 Always include a root element, root elements identify to your machines what sort of object they are consuming
24. Random meta data { "people": [{"id":1}, {"id":2}, {"id":3}], count: 3, active_count: 1 page: 1 total: 10 } Don't do this! Wednesday, 15 August 12 Please don't include counts in your response, they are not needed. Arrays provide this natively. If you need to provide paging, use the Link header
25. How do I represent X operation? Wednesday, 15 August 12 So, once we went beyond creating and destroying records in our database we had to create endpoints for starting and stopping resources, this is a very nonstandard use of REST, it is akin to providing endpoints to control a toaster — "I can't turn on unless I have bread etc"
26. GET /instances List all Instances POST /instances Create an Instance GET /instances/:id Fetch an Instance HTTP METHODS Wednesday, 15 August 12 Lets talk about methods, HTTP has a bunch of them, you've probably all seen these ones, they have become pretty much the standard interface of the web these days
27. HTTP METHODS GET /instances List all Instances POST /instances Create an Instance GET /instances/:id Fetch an Instance PUT /instances/:id Replace an Instance DELETE /instances/:id Destroy all Instance PATCH /instances/:id Update an Instance HEAD /instances/:id Return headers only OPTIONS /instances Supported methods TRACE /instances Echo CONNECT /instances Mad Hax Wednesday, 15 August 12 But what about these ones, defined in RFC 2116, surely we could make use of a few of these.
28. STATUS GET /instances 200 OK POST /instances 201 Created GET /instances/:id 200 OK PUT /instances/:id 200 OK DELETE /instances/:id 204 No Content PATCH /instances/:id 200 OK Status codes tell machines how to handle the response. Wednesday, 15 August 12 You may have noticed on my previous slide about testing that I was asserting the correct status code was being returned, this is important as it tells machines how they should handle the response.
29. Lock it down Strategies: ‣ Token+Secret ‣ HMAC ‣ HTTP Basic/Digest ‣ OAuth / OAuth2 ‣ X-Auth ‣ Multi-factor SSL shared key Wednesday, 15 August 12 If your method for testing the API you're building involved reloading curl or a web browser, you are doing things seriously wrong. With a few simple tools you can test the JSON responses from you API with very little code, and I'll show you an example at the end.
30. Extra Reading: ‣ Grape Entities ‣ Grape Presenters ‣ Using RABL with grape ‣ Devise for authentication ‣ HATEOAS (Hypermedia as the Engine of Application State) Special thanks to github.com/intridea/grape Wednesday, 15 August 12 - versioning - defining endpoints - rabl templates - presenting resources with Rabl - testing endpoints - Authentication Closing thoughts: - Error handling - Response formats - Grape presenters - helpers - Using with devise