How a Robot Stole Our Jobs (and did it better)

Transcript

1. a HOW A ROBOT STOLE OUR JOBS (AND DID IT BETTER)
2. NO PHOTOS
3. CARLOS DIAZ-PADRON | @CDPADRON Carlos Diaz-Padron API Team
4. THE API TEAM - POST-YOYODYNE ‣ Manage and develop: ‣ Public API proxy layer and resources ‣ Public REST API helper libraries CARLOS DIAZ-PADRON | @CDPADRON
5. API TEAM PERSPECTIVE - PRODUCT PIPELINE CARLOS DIAZ-PADRON | @CDPADRON Request from Product Team Server-Side Resources Integration/Cluster Tests Library Development Deploy
6. CARLOS DIAZ-PADRON | @CDPADRON INITIAL STRATEGY: WHAT DO WE HAVE NOW? PYTHON PHP C# JAVA RUBY NODE JS
7. CARLOS DIAZ-PADRON | @CDPADRON INITIAL STRATEGY: WHAT DO WE HAVE NOW? PYTHON PHP C# JAVA RUBY NODE JS
8. THE SEARCH FOR EXISTING SOLUTIONS ‣ Swagger ‣ Swagger (now Open API) Specification is too closely tied to concepts in HTTP ‣Very low-level, makes no assumptions about well-formed RESTful resources ‣ The resultant library artifacts are a little clunky to use ‣ gRPC ‣ Also very low level and tied to HTTP ‣Optimized for more complex logic, whereas our REST API is entirely stateless and doesn’t use many bells and whistles of HTTP CARLOS DIAZ-PADRON | @CDPADRON
9. CARLOS DIAZ-PADRON | @CDPADRON
10. WHAT DO WE NEED? ‣ Declarative way to define our API ‣ Any logic in the definitions of resources makes future refactoring a huge pain ‣ Leaves little room for bad practices in definitions ‣ Generate helper libraries, automatically, from these declarative definitions ‣ Definitions need to be as small as possible, conventional, and easy for a non-engineer to develop CARLOS DIAZ-PADRON | @CDPADRON
11. WHAT ELSE DO WE NEED? ‣ Strong integration test suite ‣ Definitions powerful enough to be able to express any future APIs ‣But restrictive enough to discourage bad practices CARLOS DIAZ-PADRON | @CDPADRON
12. SOLUTION: YOYODYNE ‣ Code generator that works off of declarative definitions, expressed in JSON ‣ Designed to derive a lot from a little information ‣ Every future change to helper libraries should require just 1 set of changes to API definitions ‣ No work should be required for each language CARLOS DIAZ-PADRON | @CDPADRON
13. YOYODYNE ARCHITECTURE CARLOS DIAZ-PADRON | @CDPADRON Definitions Python Controller Java Controller … Python Views Java Views Artifacts Definitions Definitions
14. CARLOS DIAZ-PADRON | @CDPADRON YOYODYNE ARCHITECTURE: DEFINITIONS . ├── alias │ └── v2010 │ ├── call.json │ ├── message.json ├── base │ ├── api │ │ └── v2010 │ │ ├── account.json │ │ ├── call.json │ │ ├── message.json │ └── api │ └── v2010 │ ├── account.json │ ├── call.json │ ├── message.json ├── genes │ └── genes.json ├── transactions │ ├── api │ │ └── v2010 │ │ ├── account.json │ │ ├── call.json │ │ ├── message.json └── types ├── csharp.json ├── empty.json ├── java.json ├── node.json ├── php.json ├── python.json └── ruby.json
15. CARLOS DIAZ-PADRON | @CDPADRON { "class_name": "call", "mount_name": "calls", "file_name": "call", "location": { "protocol": "https", "domain": "api.twilio.com", "version": "2010-04-01", "list": "/Accounts/{account_sid}/Calls.json", "instance": "/Accounts/{account_sid}/Calls/{sid}.json" }, "enums": { "event": [ "initiated", "ringing", "answered", "completed" ], "status": [ "queued", "ringing", "in-progress", "completed", "busy", "failed", "no-answer", "canceled" ] }, "properties": { "account_sid": "sid<AC>", "annotation": "string", "answered_by": "string", "api_version": "string", "caller_name": "string", "date_created": "date_time<rfc2822>", "date_updated": "date_time<rfc2822>", "direction": "string", "duration": "string", "end_time": "date_time<rfc2822>", "forwarded_from": "string", "from": "string{OriginationPhoneNumber}", "from_formatted": "string", "group_sid": "sid<GP>", "parent_call_sid": "sid<CA>", "phone_number_sid": "sid<PN>", "price": "decimal", "price_unit": "currency", "sid": "sid<CA>", "start_time": "date_time<rfc2822>", "status": "enum:status", "subresource_uris": "uri_map", "to": "string", "to_formatted": "string", "uri": "uri" }, "actions": { "create": { "method": "post", "location": "list", "params": { "required": { "to": "phone_number", "from": "phone_number{VoiceOriginationPhoneNumber}" }, "conditional": [ { "url": "url", "application_sid": "sid<AP>" } ], "optional": { "method": "http_method", "fallback_url": "url", "fallback_method": "http_method", "status_callback": "url", "status_callback_method": "http_method", "send_digits": "string", "if_machine": "string", "timeout": "integer", "record": "boolean" }, "path": { "account_sid": "sid<AC>" } } }, "delete": { "method": "delete", "location": "instance", "params": { "path": { "account_sid": "sid<AC>", "sid": "sid<CA>" } } }, ‣ 1 for each resource (eg. Call, Message) ‣ Define: ‣ What it is ‣ What it looks like ‣ What actions can be done to it ‣ Relationships BASE DEFINITIONS
16. CARLOS DIAZ-PADRON | @CDPADRON { "class_name": "call", "mount_name": "calls", "file_name": "call", "location": { "protocol": "https", "domain": "api.twilio.com", "version": "2010-04-01", "list": "/Accounts/{account_sid}/Calls.json", "instance": "/Accounts/{account_sid}/Calls/ {sid}.json" }, ‣ Names ‣ The location in the Twilio API ‣ The location for its list and instance endpoints (if they exist) WHAT IT IS
17. CARLOS DIAZ-PADRON | @CDPADRON "enums": { "event": [ "initiated", "ringing", "answered", "completed" ], "status": [ "queued", "ringing", "in-progress", "completed", "busy", "failed", "no-answer", "canceled" ] }, "properties": { "account_sid": "sid<AC>", "annotation": "string", "answered_by": "string", "api_version": "string", "caller_name": "string", "date_created": "date_time<rfc2822>", "date_updated": "date_time<rfc2822>", "direction": "string", "duration": "string", ‣ Values for enums ‣ What properties it has ‣ What types are those properties WHAT IT LOOKS LIKE
18. CARLOS DIAZ-PADRON | @CDPADRON ‣ What type of action it is ‣ (HTTP method since this is an HTTP resource) ‣ What properties are required/optional ‣ Types ‣ What properties we need for the path WHAT ACTIONS CAN BE DONE "actions": { "create": { "method": "post", "location": "list", "params": { "required": { "to": "phone_number", "from": "phone_number{VoiceOriginationPhoneNumber}" }, "conditional": [ { "url": "url", "application_sid": "sid<AP>" } ], "optional": { "method": "http_method", "fallback_url": "url", "fallback_method": "http_method", "status_callback": "url", "status_callback_method": "http_method", "send_digits": "string", "if_machine": "string", "timeout": "integer", "record": "boolean" }, "path": { "account_sid": "sid<AC>" } }
19. CARLOS DIAZ-PADRON | @CDPADRON ‣ What type of action it is ‣ (HTTP method since this is an HTTP resource) ‣ What properties are required/optional ‣ Types ‣ What properties we need for the path RELATIONSHIPS "components": [ "api/v2010/call_feedback_summary" ], "dependents": { "api/v2010/call_recording": { "account_sid": "account_sid", "call_sid": "sid" }, "api/v2010/call_notification": { "account_sid": "account_sid", "call_sid": "sid" }, "api/v2010/call_feedback": { "account_sid": "account_sid", "call_sid": "sid" } }, "parent": “api/v2010/account" }
20. CARLOS DIAZ-PADRON | @CDPADRON ‣ What language type to use for each semantic type TYPES { "properties": { "_": "String" , "boolean": "Boolean" , "currency": "Currency" , "date": "LocalDate" , "date_time": "DateTime" , "date_time_range": "Range<DateTime>" , "decimal": "BigDecimal" , "http_method": "HttpMethod" , "integer": "Integer" , "url": "URI" , "uri_map": "Map<String, String>" , "string_map": "Map<String, String>" , "phone_number": "com.twilio.sdk.type.PhoneNumber" , "ice_server": "IceServer" , "phone_number_capabilities": "PhoneNumberCapabilities" , "feedback_issue": "FeedbackIssue" , "object": "JsonNode" , "phone_number_price": "PhoneNumberPrice" , "outbound_sms_price": "OutboundSmsPrice" , "inbound_sms_price": "InboundSmsPrice" , "outbound_prefix_price": "OutboundPrefixPrice" , "inbound_call_price": "InboundCallPrice" , "outbound_call_price": "OutboundCallPrice" },
21. CARLOS DIAZ-PADRON | @CDPADRON ‣ Actual recorded interactions for the API ‣ Used for integration tests ‣ Satisfies the “air gap” TRANSACTIONS { "create": [ { "name": "create", "request": { "account_sid": "ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "form_params": { "ApplicationSid": "APaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "FallbackMethod": "GET", "FallbackUrl": "https://example.com", "From": "+987654321", "IfMachine": "if_machine", "Method": "GET", "Record": "true", "SendDigits": "send_digits", "StatusCallback": "https://example.com", "StatusCallbackMethod": "GET", "Timeout": 1, "To": "+123456789", "Url": "https://example.com" }, "method": "POST", "query_params": {}, "url": "https://api.twilio.com/2010-04-01/Accounts/ACaaaaaaaaaa }, "response": { "content": { "account_sid": "ACaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "annotation": null, "answered_by": null, "api_version": "2010-04-01", "caller_name": null, ", "status": "completed",
22. CARLOS DIAZ-PADRON | @CDPADRON { "filename": "available_phone_numbers/local.rb", "symbol": "local", "list_name": "Local", "instance_class": "Twilio::REST::AvailablePhoneNumber" } ‣ Define the “weirdness” in existing libraries COMPATIBILITY DEFINITIONS
23. COMPATIBILITY DEFINITIONS: PROBLEM ‣ This now violates our original intention with Yoyodyne ‣ 1 change to API = 1 base definition + a compatibility per language ‣ Compatibility definitions have made framework more complicated ‣ It’s time to re-group and re-think our approach CARLOS DIAZ-PADRON | @CDPADRON
24. SECOND TIME’S THE CHARM ‣ Avoided making breaking changes to make our libraries ‣ Generating the current libraries is becoming more work than manually developing them ‣ Breaking changes must be made CARLOS DIAZ-PADRON | @CDPADRON
25. NEW STRATEGY: MAKE EVERYTHING BETTER ‣ Common pain points: ‣ Inconsistent resources within libraries ‣ Paging ‣ Error handling ‣ Efficiency: Reducing request load on API by introducing assumptions on its behavior (eg. pre-emptive auth) ‣ Future proofing: Subdomains as first-class citizens ‣Internal Structure, more conducive to auto-generation CARLOS DIAZ-PADRON | @CDPADRON
26. CARLOS DIAZ-PADRON | @CDPADRON $calls = $client->account->calls->list(); while ($calls) { foreach ($calls as $call) { // Something } $calls = calls->next_page(); } foreach ($client->account->calls->stream() as $call) { print($call->sid); } BEFORE AFTER
27. API TEAM PERSPECTIVE - PRODUCT PIPELINE CARLOS DIAZ-PADRON | @CDPADRON Request from Product Team Server-Side Resources Integration/Cluster Tests Create Yoyodyne Definitions Deploy
28. CARLOS DIAZ-PADRON | @CDPADRON CLASS MAKECALL: METHOD = “POST” FIELDS = [ REQUIREDFIELD(‘TO’, TYPE=‘STRING’), … ] ‣ Most API resources are declaratively defined in Python code within Starship, our API proxy ‣ Stateless with transformations Sam Kimbrel is giving a talk about Starship’s architecture tomorrow at 4:15pm! DPR: DECLARATIVE PROXIED RESOURCE
29. CARLOS DIAZ-PADRON | @CDPADRON
30. CPR: CONFIGURED PROXIED RESOURCE ‣ The Yoyodyne base definitions have almost everything we need to generate DPRs for resources: ‣ We know what a resource looks like ‣ We know what we can do with a resource and how we can do it CARLOS DIAZ-PADRON | @CDPADRON
31. CARLOS DIAZ-PADRON | @CDPADRON { "actions": { "read": { "service_base_uri": “wireless-service", "beta_feature": “api.beta-flag“, "request_transforms": [ { "name": "request_path_transform", "args": { "path": “/v1/Devices/{id}" } } ], "response_transforms": [ ] } } } ‣ Missing is how we interact with the downstream service ‣ Define how we communicate to downstream services to get results for the user CPR DEFINITIONS
32. API TEAM PERSPECTIVE - PRODUCT PIPELINE CARLOS DIAZ-PADRON | @CDPADRON Base Definitions from Product Team Deploy Integration/Cluster Tests CPRs/Libraries
33. CARLOS DIAZ-PADRON | @CDPADRON { "VoiceOriginationPhoneNumber": { "api/v2010/incoming_phone_number": "phone_number", "api/v2010/outgoing_caller_id": "phone_number" }, "OriginationPhoneNumber": { "api/v2010/incoming_phone_number": "phone_number", "api/v2010/incoming_phone_number_local": "phone_number", "api/v2010/incoming_phone_number_mobile": "phone_number", "api/v2010/incoming_phone_number_toll_free": "phone_number" }, "AvailablePhoneNumber": { "api/v2010/available_phone_number_local": "phone_number", "api/v2010/available_phone_number_mobile": "phone_number", "api/v2010/available_phone_number_toll_free": "phone_number" }, "AvailableLocalNumber": { "api/v2010/available_phone_number_local": "phone_number" }, "AvailableMobileNumber": { "api/v2010/available_phone_number_mobile": "phone_number" }, "AvailableTollFreeNumber": { "api/v2010/available_phone_number_toll_free": "phone_number" }, "AvailableNumberCountry": { "api/v2010/available_phone_number_country": "country_code" }, "CredentialList": { "api/v2010/sip_credential_list": "sid" }, "IpAccessControlList": { "api/v2010/sip_ip_access_control_list": "sid" }, ‣ The base definitions describe the relationships between parent and sub resources ‣ Genes define where Twilio-derived parameters come from AUXILIARY DEFINITION: GENES
34. CARLOS DIAZ-PADRON | @CDPADRON AUXILIARY DEFINITION: GENES "actions": { "create": { "method": "post", "location": "list", "params": { "required": { "to": "phone_number", "from": "phone_number{VoiceOriginationPhoneNumber}" }, "conditional": [ { "url": "url", "application_sid": "sid<AP>" } ], "optional": { "method": "http_method", "fallback_url": "url", "fallback_method": "http_method", "status_callback": "url", "status_callback_method": "http_method", "send_digits": "string", "if_machine": "string", "timeout": "integer", "record": "boolean" }, "path": { "account_sid": "sid<AC>" } } },
35. CARLOS DIAZ-PADRON | @CDPADRON FULLY AUTOMATIC AVAILABILITY TESTS WITH “HUBBLE” ‣ Hubble now knows how to generate all fixtures ‣ Resource information is cached in Runscope
36. CARLOS DIAZ-PADRON | @CDPADRON
37. CLUSTER TESTS ‣ Use that dependency graph to build test fixtures ‣ Automatically Generated Cluster Tests CARLOS DIAZ-PADRON | @CDPADRON
38. API TEAM PERSPECTIVE - PRODUCT PIPELINE CARLOS DIAZ-PADRON | @CDPADRON Base Definitions from Product Team Deploy Generate Availability Tests CPRs/Libraries/Tests
39. CARLOS DIAZ-PADRON | @CDPADRON BEFORE: /Devices/{id}/RatePlan AFTER: /RatePlan/{id} ‣ Wireless API was designed to have the RatePlan resource deeply nested in the Device instance ‣ Yoyodyne didn’t like it because RatePlan doesn’t have a `device_sid` CASE STUDY: WIRELESS RATEPLANS
40. CASE STUDY: SIGNAL RELEASE PRODUCTS CARLOS DIAZ-PADRON | @CDPADRON Add 1 new API, with Starship
 and Library support pre-Yoyodyne Generate CPRs for 3 APIs, including new Wireless API, and generate the libraries with Yoyodyne 1-4 weeks
 including UAT, debugging, and fixes 1 day
41. THE API TEAM - POST-YOYODYNE ‣ Manage and develop: ‣ Public API proxy layer and resources ‣ Public REST API helper libraries CARLOS DIAZ-PADRON | @CDPADRON ‣ Manage and develop: ‣ Public API proxy layer and resources ‣ Public REST API helper libraries
42. Carlos Diaz-Padron carlosdp@cmu.edu (305) 815-0068 Carnegie Mellon University SMC 5595 Pittsburgh, PA 15213 Website/Blog: http://carlosdp.io – Github: https://github.com/carlosdp Education • Carnegie Mellon University, May 2015 ◦ Bachelor of Science in Information Systems, emphasis Computer Science ◦ Relevant course work: Intro to Computer Systems, Web Application Development, Distributed Systems, Compilers (F14) Skills • Polyglot, experienced with enough languages to be able to pick a new one up quickly • Fluent in Ruby (and Rails), Java, SQL, HTML/CSS, Javascript (and popular frameworks), PHP, Go • Experience with Python, C#, Bash, C/(some C++) • Experienced with version control and very capable with Git advanced features • Experience (and love for) Test-Driven Development Experience Mozilla Corporation (Summer 2014) Software Engineer Intern • Worked on the Cloud Services team on originating the BackgroundSync API for Firefox OS / Firefox Android. • Interfaced with teams in Mozilla and Google Chromium to help develop a standard specification. Twilio, Inc. (Summer 2013) Software Engineer Intern • Worked on Voice Team developing metrics solutions for tracking call quality for Twilio's voice products (mainly in Java). • Worked with API team on open-source maintenance and currently maintain the twilio-ruby library. Carnegie Mellon Student Government (Aug 2011- Aug 2013) Applications Engineer • Maintained and worked on several web applications that manage student organizations and budgets. • Upgraded elections software to host CMU's first ever electronic nominations for student government. Projects (Open Source contributions and projects available at http://github.com/carlosdp) twiliogo (https://github.com/carlosdp/twiliogo) Twilio helper library for the Go programming language Scotch (http://snstheatre.org) Theatrical Management Solution that handles conflict-aware scheduling, communication, and patron tracking myUniSchedule (www.myunischedule.com) Shows CMU/UPenn students who is in their classes by sharing schedules using Facebook OAuth, developed in 24 hours summer before freshman year Leadership Scotch'n'Soda Theatre (CMU) Managing Director (February 2013 – present) Production Manager, 12 Angry Men (Feb. 2013), Judas Iscariot (Feb. 2014), How to Succeed (Oct. 2014)
43. API TEAM PERSPECTIVE - PRODUCT PIPELINE CARLOS DIAZ-PADRON | @CDPADRON Base Definitions from Product Team Deploy Generate Availability Tests CPRs/Libraries/Tests
44. THE API TEAM - POST-YOYODYNE ‣ Manage and develop: ‣ Framework of API proxy layer ‣ Framework of REST API helper libraries ‣ Internal Platform API ‣ Cloud resource garbage collector, Janitor ‣ Other useful tools CARLOS DIAZ-PADRON | @CDPADRON
45. CARLOS DIAZ-PADRON | @CDPADRON CARLOS DIAZ-PADRON @CDPADRON http://bit.ly/helper-libs