MongoDB and REST APIs A Match Made in Heaven

Slide 1

Slide 1 text

Good Morning Percona Live Amsterdam 2015

Slide 2

Slide 2 text

Who Am I fighting back against impostor syndrome

Slide 3

Slide 3 text

Nicola Iarocci Co-founder and CTO at CIR 2000

Slide 4

Slide 4 text

Nicola Iarocci MongoDB Master

Slide 5

Slide 5 text

Nicola Iarocci Open Source junkie Eve • Cerberus • Events • Flask-Sentinel • Eve.NET • Etc.

Slide 6

Slide 6 text

Nicola Iarocci Consultant Mongo • RESTful Services • Python • My Open Source Projects

Slide 7

Slide 7 text

Nicola Iarocci CoderDojo Coding Clubs for Kids

Slide 8

Slide 8 text

MongoDB & REST APIs A Match Made in Heaven

Slide 9

Slide 9 text

Agenda

Slide 10

Slide 10 text

Agenda 1. Our use case for a RESTful API

Slide 11

Slide 11 text

Agenda 2. What is a RESTful API and why we need it

Slide 12

Slide 12 text

Agenda 3. Why MongoDB is a good match for RESTful Services

Slide 13

Slide 13 text

Agenda 4. Build and run a MongoDB RESTful Service from scratch, live on stage.

Slide 14

Slide 14 text

Agenda 5. Stories from the field (if there’s any time left, which I doubt)

Slide 15

Slide 15 text

The Case for RESTful Web APIs

Slide 16

Slide 16 text

Amica 10 invoicing & accounting for italian small businesses

Slide 17

Slide 17 text

your old school desktop app

Slide 18

Slide 18 text

what we started with Client LAN/SQL Database Desktop Application

Slide 19

Slide 19 text

Goal A remote service that client apps can leverage to stay in sync withc each other

Slide 20

Slide 20 text

What we need #1 Must be accessible by any kind of client technology

Slide 21

Slide 21 text

What we need #2 Abstract the data access layer so we can update/replace the engine at any time with no impact on clients

Slide 22

Slide 22 text

What we need #3 An appropriate data storage engine

Slide 23

Slide 23 text

What we need #4 Easily (re)deployable and scalable multi-micro-service architecure

Slide 24

Slide 24 text

Where we want to go Clients “Cloud” Database RESTful Web API API iOS Android Website Desktop Client ? ?

Slide 25

Slide 25 text

Constraints • minimum viable product first • add features over time • frequent database schema updates • easily scalable • avoid downtime as much as possible • cater upfront for a microservices architecture

Slide 26

Slide 26 text

REST So What Is REST All About?

Slide 27

Slide 27 text

REST is not a standard

Slide 28

Slide 28 text

REST is not a protocol

Slide 29

Slide 29 text

REST is an architectural style for networked applications

Slide 30

Slide 30 text

Defines a set of simple principles loosely followed by most API implementations

Slide 31

Slide 31 text

“resource” the source of a specific information

Slide 32

Slide 32 text

A web page is not a resource rather the representation of a resource

Slide 33

Slide 33 text

“global permanent identifier” every resource is uniquely identified. Think a HTTP URI.

Slide 34

Slide 34 text

#3 standard interface used to exchange representations of resources (think the HTTP protocol)

Slide 35

Slide 35 text

“a set of constraints” separation of concerns, stateless, cacheability, layered system, uniform interface, etc.

Slide 36

Slide 36 text

Web is built on REST and it is meant to be consumed by humans

Slide 37

Slide 37 text

RESTful APIs are built on REST and are meant to be consumed by machines

Slide 38

Slide 38 text

Representational State Transfer (REST) by Roy Thomas Fielding http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm

Slide 39

Slide 39 text

Goals #1 and #2 are met REST layer allows all kinds of client technologies and abstracts the data away

Slide 40

Slide 40 text

MongoDB and REST Or why we picked MongoDB for our REST API

Slide 41

Slide 41 text

JSON transport Most REST services and clients produce and consume use JSON

Slide 42

Slide 42 text

JSON-style data store MongoDB stores data as Binary JSON

Slide 43

Slide 43 text

JSON & RESTful API JSON accepted media type Client JSON (BSON) Mongo GET maybe we can push directly to client?

Slide 44

Slide 44 text

JSON & RESTful API JSON accepted media type Client JSON (BSON) Mongo JSON subset of python dict (kinda) API GET almost.

Slide 45

Slide 45 text

JSON & RESTful API JSON objects Client JSON (BSON) Mongo JSON/dict maps to python dict (validation layer) API POST also works when sending data the database

Slide 46

Slide 46 text

Similarity with RDBMS makes NoSQL easy to grasp (even for a sql head like me)

Slide 47

Slide 47 text

Terminology RDBMS Mongo Database Database Table Collection Rows(s) JSON Document Index Index Join Embedding & Linking

Slide 48

Slide 48 text

What about Queries? Queries in MongoDB are represented as JSON-style objects db.things.find({x: 3, y: "foo”});

Slide 49

Slide 49 text

Filtering and Sorting native Mongo query syntax Client JSON (BSON) Mongo (very) thin parsing & validation layer API Expose the native MongoDB syntax? ?where={“x”: 3, “y”: “foo”}

Slide 50

Slide 50 text

JSON all along the pipeline mapping to and from the database feels more natural

Slide 51

Slide 51 text

No need for ORM No need to map objects to JSON and vice-versa (win!)

Slide 52

Slide 52 text

schema-less dynamic documents allow for painless evolution

Slide 53

Slide 53 text

REST is stateless MongoDB lacks transactions

Slide 54

Slide 54 text

Ideal API Surface Mongo collection maps to API resource endpoint api.example.com/contacts Maps to a Mongo collection

Slide 55

Slide 55 text

Ideal API Surface Mongo document maps to a API document endpoint api.example.com/contacts/4f46445fc88e201858000000 Maps to a collection ObjectID

Slide 56

Slide 56 text

Goal #3 is met An appropriate data storage engine: MongoDB

Slide 57

Slide 57 text

Eve REST API for Humans™ Free and Open Source Powered by MongoDB and Good Intentions eve

Slide 58

Slide 58 text

Philosopy effortlessly build and deploy highly customizable, fully featured RESTful Web Services

Slide 59

Slide 59 text

Quickstart

Slide 60

Slide 60 text

install $ pip install eve

Slide 61

Slide 61 text

run.py from eve import Eve app = Eve() if __name__ == '__main__': app.run()

Slide 62

Slide 62 text

settings.py # just a couple API endpoints with no custom schema or rules. DOMAIN = { ‘people’: {} ‘works’: {} }

Slide 63

Slide 63 text

launch $ python run.py * Running on http://127.0.0.1:5000/

Slide 64

Slide 64 text

enjoy HATEOAS AT WORK HERE $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}

Slide 65

Slide 65 text

enjoy CLIENTS CAN EXPLORE THE API PROGRAMMATICALLY $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": “home”} }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}

Slide 66

Slide 66 text

$ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }} enjoy AND EVENTUALLY FILL THE UI

Slide 67

Slide 67 text

enjoy $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }} PAGINATION DATA

Slide 68

Slide 68 text

enjoy EMTPY RESOURCE AS WE DID NOT CONNECT A DATASOURCE $ curl http://localhost:5000/people { "_items": [], "_links": { "self": {"href": "people", "title": "people"}, "parent": {“href": "/", "title": "home"}, }, "_meta": { "max_results": 25, "total": 0, "page": 1 }}

Slide 69

Slide 69 text

settings.py # let’s connect to a mongo instance MONGO_HOST = 'localhost' MONGO_PORT = 27017 MONGO_USERNAME = 'user' MONGO_PASSWORD = 'user' MONGO_DBNAME = 'percona'

Slide 70

Slide 70 text

settings.py # let’s also add a few validation rules DOMAIN['people']['schema'] = { 'name': { 'type': 'string’, 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} UNIQUE STRING, MAX LENGTH 50

Slide 71

Slide 71 text

settings.py # let’s also add a few validation rules DOMAIN['people']['schema'] = { 'name': { 'type': 'string', 'maxlength': 50, 'unique': True}, 'email': { 'type': 'string', 'regex': '^\S+@\S+$'}, 'location': { 'type': 'dict', 'schema': { 'address': {'type': 'string'}, 'city': {'type': 'string’}}}, 'born': {'type': 'datetime'}} ONLY ACCEPT VALID EMAILS

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Slide 75

Slide 75 text

settings.py # allow write access to API endpoints # (default is [‘GET’] for both settings) # /people RESOURCE_METHODS = ['GET','POST'] # /people/ ITEM_METHODS = ['GET','PATCH','PUT','DELETE'] ADD/CREATE ONE OR MORE ITEMS

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Slide 78

Slide 78 text

Slide 79

Slide 79 text

settings.py CLIENT UI # a few optional config options DOMAIN['people'].update( { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )

Slide 80

Slide 80 text

settings.py CLIENT CACHE OPTIONS # a few optional config options DOMAIN['people'].update( { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )

Slide 81

Slide 81 text

settings.py ALTERNATE ENDPOINT # a few optional config options DOMAIN['people'].update( { 'item_title': 'person', 'cache_control':'max-age=10,must-revalidate', 'cache_expires': 10, 'additional_lookup': { 'url’: 'regex("[\w]+")', 'field’: 'name'} } } )

Slide 82

Slide 82 text

Features Overview We are going to focus on MongoDB power-ups

Slide 83

Slide 83 text

full range of CRUD operations Create/POST Read/GET Update/PATCH and Replace/PUT Delete/DELETE

Slide 84

Slide 84 text

filters, mongo style ?where={“name”: “john”}

Slide 85

Slide 85 text

filters, the python way ?where=name==john

Slide 86

Slide 86 text

sorting ?sort=city,-name SORT BY CITY, THEN NAME DESCENDING

Slide 87

Slide 87 text

projections ?projection={"avatar": 0} RETURN ALL FIELDS BUT ‘AVATAR’

Slide 88

Slide 88 text

projections ?projection={"lastname": 1} ONLY RETURN ‘LASTNAME’

Slide 89

Slide 89 text

pagination ?max_results=20&page=2 MAX 20 RESULTS PER PAGE; PAGE 2

Slide 90

Slide 90 text

GeoJSON support and validation for all GeoJSON types Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, GeometricalCollection

Slide 91

Slide 91 text

document embedding joins

Slide 92

Slide 92 text

standard request $ curl example.com/works/ {  "title": "Book Title",  "description": "book description",  "author": “52da465a5610320002660f94"  } RAW FOREIGN KEY (DEFAULT)

Slide 93

Slide 93 text

request an embedded document $ curl example.com/works/?embedded={“author”: 1} {  "title": "Book Title",  "description": "book description",  "author": {  “firstname”: “Mark”,  “lastname”: “Green”,  }  } REQUEST EMBEDDED AUTHOR

Slide 94

Slide 94 text

embedded document $ curl example.com/works/?embedded={“author”: 1} {  "title": "Book Title",  "description": "book description",  "author": {  “firstname”: “Mark”,  “lastname”: “Green”,  }  } # embedding is configurable on per-field basis and # can be pre-set by API maintainer EMBEDDED DOCUMENT

Slide 95

Slide 95 text

bulk inserts insert multiple documents with a single request

Slide 96

Slide 96 text

request $ curl -d ‘ [ { "firstname": "barack", "lastname": “obama" }, { "firstname": "mitt", "lastname": “romney” } ]' -H 'Content-Type: application/json’

Slide 97

Slide 97 text

response [ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “”, "title": "person"}} }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “", "title": "person"}} } ] COHERENCE MODE OFF: ONLY META FIELDS ARE RETURNED

Slide 98

Slide 98 text

response [ { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5b", "_etag": "749093d334ebd05cf7f2b7dbfb7868605578db2c" "_links": {"self": {"href": “”, "title": “person”}}, "firstname": "barack", "lastname": "obama", }, { "_status": "OK", "_updated": "Thu, 22 Nov 2012 15:22:27 GMT", "_id": "50ae43339fa12500024def5c", "_etag": "62d356f623c7d9dc864ffa5facc47dced4ba6907" "_links": {"self": {"href": “", "title": "person"}} "firstname": "mitt", "lastname": "romney", } ] COHERENCE MODE ON: ALL FIELDS RETURNED

Slide 99

Slide 99 text

document versioning ?version=3 ?version=all ?version=diffs

Slide 100

Slide 100 text

soft deletes preserve deleted documents and retrieve them with a simple ?show_deleted

Slide 101

Slide 101 text

file storage files are stored in GridFS by default; customizable for S3, file system, etc.

Slide 102

Slide 102 text

data validation powerful and extensible data validation powered by Cerberus

Slide 103

Slide 103 text

rich validation grammar referentrial integrity / unique values / defaults / regex / etc.

Slide 104

Slide 104 text

custom data types create your own data types to validate against

Slide 105

Slide 105 text

custom validation logic extended the validation system to cater for specific use cases

Slide 106

Slide 106 text

multi database serve endpoints and/or users from dedicated mongos

Slide 107

Slide 107 text

index maintenance define sets of resource indexes to be (re)created at launch supports sparse, geo2d and background indexes

Slide 108

Slide 108 text

event hooks plug custom actions in the request/response cycle

Slide 109

Slide 109 text

run.py from eve import Eve app = Eve() def percona(resource, response): documents = response['_items'] for document in documents: document['percona'] = 'is so cool!' if __name__ == '__main__': app.on_fetched_resource += percona_live app.run() CALLBACK FUNCTION

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Slide 112

Slide 112 text

Slide 113

Slide 113 text

rate limiting powered

Slide 114

Slide 114 text

settings.py # Rate limit on GET requests: RATE_LIMIT_GET = (1, 60) ONE REQUEST PER MINUTE (CLIENT)

Slide 115

Slide 115 text

$ curl -i HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1390486659 rate limited request CURRENT LIMIT

Slide 116

Slide 116 text

$ curl -i HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1390486659 rate limited request REMAINING

Slide 117

Slide 117 text

$ curl -i HTTP/1.1 200 OK X-RateLimit-Limit: 1 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1390486659 rate limited request TIME TO RESET

Slide 118

Slide 118 text

$ curl -i HTTP/1.1 429 TOO MANY REQUESTS rate limited request OUCH!

Slide 119

Slide 119 text

operations log log all operations and eventually expose a dedicated endpoint

Slide 120

Slide 120 text

HATEOAS Hypermedia As Engine Of The Application State

Slide 121

Slide 121 text

XML support $ curl -H ”Accept: application/xml” -i http://example.com/people <_created>Fri, 18 Sep 2015 13:41:37 GMT <_etag>5d057712ce792ebb4100b96aa98bfe9b6693c07b <_id>55fc149138345b0880f07e3d <_updated>Fri, 18 Sep 2015 13:41:37 GMT johndoe@gmail.com john

Slide 122

Slide 122 text

conditional requests allow clients to only request non-cached content

Slide 123

Slide 123 text

If-Modified-Since If-Modified-Since: Wed, 05 Dec 2012 09:53:07 GMT ONLY RETURN DOCUMENT IF MODIFIED SINCE

Slide 124

Slide 124 text

If-None-Match If-None-Match:1234567890123456789012345678901234567890 > ONLY RETURN DOCUMENT ETAG CHANGED

Slide 125

Slide 125 text

data integrity and concurrency no overwriting documents with obsolete versions

Slide 126

Slide 126 text

missing ETag # fails, as there is no ETag included with request $ curl \ -X PATCH \ -i http://example.com/people/521d6840c437dc0002d1203c \ -H "Content-Type: application/json" \ -d '{"name": “ronald"}' HTTP/1.1 403 FORBIDDEN NO ETAG REJECTED

Slide 127

Slide 127 text

ETag mismatch # fails, as ETag does not match with server $ curl \ -X PATCH \ -i http://example.com/people/521d6840c437dc0002d1203c \ -H "If-Match: 1234567890123456789012345678901234567890" \ -H "Content-Type: application/json” \ -d '{"firstname": "ronald"}' HTTP/1.1 412 PRECONDITION FAILED ETAG MISMATCH REJECTED

Slide 128

Slide 128 text

valid ETag # success at last! ETag matches with server $ curl \ -X PATCH \ -i http://example.com/people/50adfa4038345b1049c88a37 \ -H "If-Match: 80b81f314712932a4d4ea75ab0b76a4eea613012" \ -H "Content-Type: application/json" \ -d '{"firstname": "ronald"}' HTTP/1.1 200 OK # Like most of features, ETags can be disabled. ETAG MATCH ACCEPTED

Slide 129

Slide 129 text

custom data layers build your own data layer

Slide 130

Slide 130 text

authentication and authorization basic / token / hmac / BYO / OAuth2 / you name it

Slide 131

Slide 131 text

and (a lot) more CORS, cache control, API versioning, JOSNP, Etc.

Slide 132

Slide 132 text

vibrant community 90+ contributors / 350+ forks / 2500+ github stargazers

Slide 133

Slide 133 text

Eve-Docs automatic documentation for Eve APIs in both HTML and JSON CHARLES FLYNN

Slide 134

Slide 134 text

Eve-Docs

Slide 135

Slide 135 text

Eve-Elastic Elasticsearch data layer for your Eve-powered API PETR JASEK

Slide 136

Slide 136 text

Eve-SQLAlchemy SQL data layer for Eve-powered APIs PETR JASEK

Slide 137

Slide 137 text

Eve-Mongoengine enables mongoengines data models to be used as Eve schema STANISLAV HELLER

Slide 138

Slide 138 text

Eve.NET HTTP and REST client for Eve-powered APIs PETR JASEK

Slide 139

Slide 139 text

Eve-OAuth2 leverage Flask-Sentinel to protect your API endpoints with OAuth2 THOMAS SILEO

Slide 140

Slide 140 text

REST Layer “golang REST API framework heavily inspired by Python Eve” THOMAS SILEO

Slide 141

Slide 141 text

Goal # 4 achieved easy to setup, launch and scale up; also a good fit for microservices infrastracture

Slide 142

Slide 142 text

A look back to initial draft Clients “Cloud” Database RESTful Web API API iOS Android Website Desktop Client ? ?

Slide 143

Slide 143 text

Clients Multiple MongoDBs Database Adam eve instances API iOS Android Website Desktop Client what we have in production

Slide 144

Slide 144 text

stories from the trenches #1. when too much (magic) is too much #2. sometimes you don’t want to debug #3. so how do I login into this thing?

Slide 145

Slide 145 text

Take Aways

Slide 146

Slide 146 text

Enhance MongoDB with powerful features on top of native engine validation, document embedding (joins), referential integrity, document versioning, transformations, rate limiting, etc.

Slide 147

Slide 147 text

Consider the REST layer as an ideal data access layer the story of pymongo 3.0 breaking changes mongo or sql or elastic or …

Slide 148

Slide 148 text

Consider the REST layer as an ideal data access layer the story of Adam dashboards

Slide 149

Slide 149 text

Consider Microservices leverage Eve features create a network of isolated yet standardized services   each service has a dedicated role runs as an eve instance, with its own configuration has its own database(s) callbacks route traffic between services

Slide 150

Slide 150 text

Clients User-reserved MongoDBs eve-multidb Data Auth eve-oauth2 (ﬂask-sentinel) API iOS Android Website Desktop Client Adam 1 Adam eve instance Redis auth tokens, rate limiting Auth/Users MongoDB

Slide 151

Slide 151 text

Clients service- reserved MongoDBs Data Auth eve-oauth2, ﬂask-sentinel API iOS Android Website Desktop Client Adam 2 Adam eve instance Redis auth tokens, rate limiting Services eve instances Auth/Users MongoDB very simplified

Slide 152

Slide 152 text

thanks nicolaiarocci python-eve.org eve