Rock Solid REST APIs

Rock Solid REST APIs

by Bob Binder @ APIStrat 2014 in Chicago

Transcript

  1. 20140909 System Verification Associates © 2014 1 How to Release

    Rock-solid RESTful APIs API Strategy and Practice Chicago, September 26, 2014 Robert V. Binder System Verification Associates Enabling High Assurance http://sysverif.com
  2. 20140909 System Verification Associates © 2014 2 Overview • How

    to validate 50,000 pages of API Doc • Advanced API Verification • Dataflow Testing Model • Model-based Testing Demo • Q&A
  3. 20140909 System Verification Associates © 2014 3 Verified 500 APIs

    • Massive project • Model-based Testing • Document scrutinized • Many Plugfests http://msdn.microsoft.com/en-us/library/jj712081.aspx Windows Protocol Technical Documents
  4. 20140909 System Verification Associates © 2014 4 Discovery Analysis Design

    Verification Support ADVANCED API VERIFICATION
  5. 20140909 System Verification Associates © 2014 5 Discovery Sprint •

    Survey and catalog • API documentation • Open and closed issues • Social media views • Codebase • Usage logs • Results • Strategy • Test environment spec • Report card Discovery Analysis Design Verification Support
  6. 20140909 System Verification Associates © 2014 6 Analysis Sprint •

    Workflow • Construct usage profile • Scrutinize documentation • Abstract data model • Results • Doc issues • Gap analysis • Revised strategy Discovery Analysis Design Verification Support
  7. 20140909 System Verification Associates © 2014 7 Design Sprint •

    Workflow • Configure virtual lab • Behavior/data models • Traffic capture/parsers • Instantiate adapters • Results • Stable test environment • All-aspect test model • Revised strategy Discovery Analysis Design Verification Support
  8. 20140909 System Verification Associates © 2014 8 Verification Sprint •

    Workflow • Model checking • Generate/run test suites • Collect traffic logs • Analyze coverage • Results • All test artifacts • Test coverage report • Final report • Briefing Discovery Analysis Design Verification Support
  9. 20140909 System Verification Associates © 2014 9 Support • As

    needed • Plug Fests • Usage monitoring • CI/regression testing • Results • Continuity • Investment protection • Continuous improvement Discovery Analysis Design Verification Support
  10. 20140909 System Verification Associates © 2014 10 Discovery Analysis Design

    Verification Support DATAFLOW TESTING MODEL
  11. 20140909 System Verification Associates © 2014 11 Test Configuration Service

    App HTTP Server HTTP Client Service HTTP Server Generated Test Code Test Model REST API
  12. 20140909 System Verification Associates © 2014 12 REST Dataflow Model

    – Normal Paths alpha Defined Used Gone PUT/201 GET/200 PUT|POST/200 DELETE/200 DELETE/200 PUT|POST/200 GET/200
  13. 20140909 System Verification Associates © 2014 13 REST Dataflow Model

    – Method Errors alpha Defined Used Gone DELETE|GET/404 DELETE|GET|PUT|POST/404
  14. 20140909 System Verification Associates © 2014 14 REST Dataflow Model

    – Parameter Errors alpha Defined Used Gone PUT|POST|GET|DELETE ?garbage/400 PUT|POST|GET|DELETE ?garbage/400
  15. 20140909 System Verification Associates © 2014 15 REST Dataflow Model

    – Parameter Errors alpha Defined Used Gone PUT|POST|GET|DELETE ?garbage/400 PUT|POST|GET|DELETE ?garbage/400 http://foo.com/bar/?who=first&what=second var request = (HttpWebRequest)WebRequest.Create("http://"+pathPrefix); request.ContentType = "text/json"; request.Method = "POST"; using (var streamWriter = new StreamWriter(request.GetRequestStream())) { string json = new JavaScriptSerializer().Serialize(new { who = "first", what = "second" }); streamWriter.Write(json); }
  16. 20140909 System Verification Associates © 2014 16 REST Dataflow Model

    alpha Defined Used Gone Test Pattern: Non-Modal Class
  17. 20140909 System Verification Associates © 2014 17 Input variation, all

    sequences • Nominal values • Boundary values • Operator mutants • Fuzzing, each/all • Domain model • Pairwise selection • Sequence randomization Sounds like a lot of work!
  18. 20140909 System Verification Associates © 2014 18 Model-based Testing •

    Model-based testing tool • Microsoft Research, 2001 • Test 500 MSFT APIs, 2007-12 • Robust and stable • Visual Studio “power tool” • C# code, not cartoons • Generates standalone executable test suite
  19. 20140909 System Verification Associates © 2014 19 Demo • Synthetic

    Client • Model Program • Coordination File • Test Cases SUT Host Test Host Test Suite HTTP Server Synthetic Client Pass/Fail Synthetic Client Interface Spex Rules Spex Cord Test Modeling Test Execution Service Under Test Explore/ Generate
  20. 20140909 System Verification Associates © 2014 20 Synthetic Client •

    The test model’s view of the SUT • Static class wrapper for HTTP client • Public methods correspond to SUT’s HTTP methods and resources • Manage server-side setup/cleanup • Message serialize/deserialize • Becomes part of the executable test code assembly • Example is a stub!
  21. 20140909 System Verification Associates © 2014 21 Model Program •

    [Rule] • Determines when an action is called • Selects argument values for the action call • Computes expected results • Updates its model state as needed • Simulates environment and/or system under test
  22. 20140909 System Verification Associates © 2014 22 Cord File •

    Defines all model actions • action = Synthetic Client public method • machine • Any action sequence • Similar to regex • May use other machines • Model any use case, scenario, slice, etc. • Many options
  23. 20140909 System Verification Associates © 2014 23 What is Exploration?

    • Find all action sequences and data bindings that model program Rules and a machine allow • Search loop • Select a rule for a machine action • If enabling condition true: • Update model program state • Return expected results • Stop when all selected inputs used or size limit exceeded
  24. 20140909 System Verification Associates © 2014 24 Machine Exploration •

    Shows all possible action sequences for a machine • No data bindings • Note similarity to normal path dataflow
  25. 20140909 System Verification Associates © 2014 25 Model Program Exploration

    • Rules + machine • Rules add data bindings, expected results • Many ways to choose data values
  26. 20140909 System Verification Associates © 2014 26 Test Cases from

    an Exploration • Spex chooses exploration steps that end in accepting state • Covers all states and steps at least once
  27. 20140909 System Verification Associates © 2014 27 Generate Test Code

    • Standalone code – does not require model • Run from VS Test Explorer or command line
  28. 20140909 System Verification Associates © 2014 28 SUT Host Test

    Host Test Suite HTTP Server Synthetic Client Pass/Fail Synthetic Client Interface Spex Rules Spex Cord Test Modeling Test Execution Service Under Test Explore/ Generate
  29. 20140909 System Verification Associates © 2014 29 Test Strategy •

    Each resource path • Interleave all DUG variants • Accepting sequence • Wrong sequence • Pairwise combination • Parameters (path and value) • Mutants, nominal, edge • Security • Interleave Fuzz cases • Abuse case model • All other HTTP methods • Performance • Virtual users/test drivers • Randomize combos
  30. 20140909 System Verification Associates © 2014 30 Q & A

    rvbinder@sysverif.com #MoreModelsLessTests http://sysverif.com
  31. 20140909 System Verification Associates © 2014 31 Discovery Analysis Design

    Verification Support ETC. Say what you do, do what you say
  32. 20140909 System Verification Associates © 2014 32 Robert V. Binder

    Robert Binder is a high-assurance entrepreneur. He has developed hundreds of application systems and advanced automated testing solutions. As test process architect for Microsoft’s Open Protocol Initiative, he lead the application of model-based testing to all of Microsoft’s server-side APIs. He is the author of the definitive Testing Object-Oriented Systems: Models, Patterns, and Tools and two other books. He holds a US patent for model-based testing of mobile systems. • MS, EECS, University of Illinois at Chicago • MBA, University of Chicago • BA, University of Chicago
  33. 20140909 System Verification Associates © 2014 33 System Verification Associates

    Enabling High Assurance • Chicago- based consulting boutique • Clients are typically software development organizations for whom system failure is not an option. • We assist clients in achieving high reliability and effectiveness in their IT processes and systems. • Founded in 2009 and led by Robert V. Binder • http://sysverif.com • Advanced API Verification Datasheet • Supported Microsoft’s Open Protocols project with a team of experts; Robert Binder served process architect, leading the technical work of over 300 staff located in Redmond, China, India, and Argentina. • Assessed and improved software process at several FDA-regulated product companies, balancing quality management system compliance and Agile practices. • Developed model-based testing solutions for high- frequency trading and aerospace applications. • Helped software service and product companies articulate unique high-value messaging for innovative services. • Conducted and published the Model-based Testing User Survey of 2012 and 2014 (forthcoming.)
  34. 20140909 System Verification Associates © 2014 34 Does My API

    Suck?  Your documentation is incomplete, wrong, misleading, or just plain incomprehensible.  Users complain that coding simple use cases is just too much hassle.  Users often rely on workarounds—they FTP files instead of using your API’s getFile.  Your API is unbalanced or incomplete—you can turn something on, but not off.  Your API’s service crashes or responds with garbage when messages are out of order or contain invalid data.  Version mismatches have unpredictable results.  No one is really sure what will happen with edge cases and they don’t want to know.  Your API allows your service to be hacked with common attack vectors.  Your service supports several protocols (REST, SOAP,…) or formats (JSON, XML,…), but behavior and data isn’t consistent  Your API doesn’t provide useful feedback— good and bad input all get the same response.  Your service is so awesome that it draws traffic spikes, but then your server chokes and dies. Buggy APIs are eating the world