Beyond REST - Coursera's API Evolution

I gave this presentation at the first annual GraphQL Summit (http://graphqlsummit.com/#Saeta), describing Coursera's API systems, and our lessons learned as we scaled to over 20 million global learners, 1.5k courses, and over a hundred partners. We discuss our successes and failures, relational algebra and an application of it to REST/JSON APIs, and finally our open source API framework called Naptime.

Transcript

1. Beyond REST Coursera’s API Evolution Revisiting theoretical foundations to improve developer productivity. Brennan Saeta ! @bsaeta " saeta
5. Problems • Number of courses grew rapidly • Product needs changed: • premium course certificates • specializations • multi-university collaborations. • Growing size of engineering team
6. Scale matters! In many dimensions • Queries per second • Data set size • Engineers • Types in ontology
8. Performance Requirements • Minimize data transfer across WAN • Minimize round trips Nice-to-have’s • Caching • ETag & Conditional Request Support • Binary Serialization
10. Experience- based APIs Queryable APIs
11. Relational Algebra Meets REST/JSON APIs Projection: filter fields in response Selection: filter items in a collection Joins: related resources
12. https://api.coursera.org/api/courses.v1?q=slug&slug=machine-learning response: { "elements": [ { "courseType": "v2.ondemand", "id": "Gtv4Xb1-EeS-ViIACwYKVQ", "name": "Machine Learning", "slug": "machine-learning" } ], "linked": {}, "paging": { "total": 1 } } request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning" }
13. https://api.coursera.org/api/courses.v1?q=slug&slug=machine-learning &fields=primaryLanguages response: { "elements": [ { "courseType": "v2.ondemand", "id": "Gtv4Xb1-EeS-ViIACwYKVQ", "name": "Machine Learning", "slug": "machine-learning", "primaryLanguages": ["en"] } ], "linked": {}, "paging": { "total": 1 } } request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "primaryLanguages" }
14. https://api.coursera.org/api/courses.v1?q=slug&slug=machine-learning &fields=instructorIds&includes=instructorIds response: { "elements": [ { "courseType": "v2.ondemand", "id": "Gtv4Xb1-EeS-ViIACwYKVQ", "name": "Machine Learning", "slug": "machine-learning", ”instructorIds": [1244] } ], "linked": { "instructors.v1": [ { "id": 1244, "name": "Andrew Ng" } ] }, "paging": { "total": 1 } } request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "instructorIds", "includes": "instructorIds" }
15. https://api.coursera.org/api/courses.v1?q=slug&slug=machine-learning &fields=instructorIds,instructors.v1(title)&includes=instructorIds response: { "elements": [ { "courseType": "v2.ondemand", "id": "Gtv4Xb1-EeS-ViIACwYKVQ", "name": "Machine Learning", "slug": "machine-learning", "instructorIds": [1244] } ], "linked": { "instructors.v1": [ { "id": 1244, "name": "Andrew Ng", "title": "Associate Professor, Stanford University; Chief Scientist, Baidu; Chairman and Co-founder, Coursera" } ] }, "paging": { "total": 1 } } request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "instructorIds, instructors.v1(title)", "includes": "instructorIds" }
16. 0 200 400 600 800 Controllers vs Naptime Resources Controllers Resources
17. Naptime Key Learnings • Leverage type safety, carefully • Collections of Key-Value pairs • Opinionated framework improves communication Principle: Optimize for developer productivity.
18. Problems remained • Flattened representation difficult for clients • URL representation cumbersome • Auto-doc tools not useable
19. Problems remained • Flattened representation difficult for clients GraphQL queries are easy to read, modify, and maintain • URL representation cumbersome GraphQL queries are easy to read, modify, and maintain • Auto-doc tools not useable GraphiQL supports auto-completion and inline documentation
21. GraphQL Query: query { CoursesV1Resource { slug(slug: "machine-learning") { id } } } request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning" }
22. request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "primaryLanguages" } GraphQL Query: query { CoursesV1Resource { slug(slug: "machine-learning") { id primaryLanguages } } }
23. request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "instructorIds", "includes": "instructorIds" } GraphQL Query: query { CoursesV1Resource { slug(slug: "machine-learning") { id instructors: instructorIds { id } } } }
24. request endpoint: /api/courses.v1 request query parameters: { "q": "slug", "slug": "machine-learning", "fields": "instructorIds, instructors.v1(title)", "includes": "instructorIds" } GraphQL Query: query { CoursesV1Resource { slug(slug: "machine-learning") { id instructors: instructorIds { id title } } } }
25. Edge Tier GraphQL Assembler Profile Service Enrollments Service Catalog Service
26. Edge Tier GraphQL Assembler Profile Service Enrollments Service Catalog Service
28. Key Lessons Learned Carefully understand your problem Look to theory: stand on the shoulders of giants Plug into the ecosystem
29. Naptime is developed in the open https://github.com/coursera/naptime
30. Thank you!
31. Thank you!
32. Questions?
33. Thank you! Photo Credits: Trevor Wilson — https://unsplash.com/@clevertrevor Pablo Garcia Saldaña — https://unsplash.com/@pagsa_ Rob Bye — https://unsplash.com/@robertbye Eli Francis - https://unsplash.com/@elifrancis Ryan Hafney — https://unsplash.com/@ryanhafey Thank you: Bryan Kane & Coursera Colleagues