Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From Big Data to Mobile Data with Apache CouchDB and PouchDB at Apache: Big Data

From Big Data to Mobile Data with Apache CouchDB and PouchDB at Apache: Big Data

It’s all too easy for mobile app developers to assume that their apps will run on fast and reliable networks. The reality for end users, though, is often slow, unreliable networks with spotty coverage. What happens when the network doesn’t work, or when a device is in airplane mode? You get unhappy, frustrated users. One solution is to take an offline-first approach. An offline-first app is an app that works, without error, when there is no network connection. Offline-first apps built with Apache CouchDB and PouchDB (an open source JavaScript database) can provide better, faster user experiences by storing data locally and then synchronizing with a cloud database when a network connection is available.

Bradley Holt

May 10, 2016
Tweet

More Decks by Bradley Holt

Other Decks in Programming

Transcript

  1. Apache: Big Data
    Bradley Holt, Developer Advocate
    Tuesday, May 10, 2016
    From Big Data to Mobile Data with
    Apache CouchDB and PouchDB
    @BradleyHolt

    View full-size slide

  2. @BradleyHolt

    View full-size slide

  3. IBM Cloud Data Services
    Open for Data
    A comprehensive por.olio of open source data services

    View full-size slide

  4. Big Data
    Get it?

    View full-size slide

  5. Why offline first?

    View full-size slide

  6. Mobile First
    Design for the smallest device first and then apply progressive enhancement techniques to
    take advantage of larger screen sizes when available

    View full-size slide

  7. Offline First
    Design for offline usage first and then apply progressive enhancement techniques to take
    advantage of network connectivity when available

    View full-size slide

  8. Ubiquitous Connectivity
    Why offline first in a world of ubiquitous connectivity?

    View full-size slide

  9. The Eight Fallacies of Distributed Computing
    1.  The network is reliable
    2.  Latency is zero
    3.  Bandwidth is infinite
    4.  The network is secure
    5.  Topology doesn't change
    6.  There is one administrator
    7.  Transport cost is zero
    8.  The network is homogeneous
    @BradleyHolt

    View full-size slide

  10. Mobile Backend
    What happens when your mobile backend service is unreachable?

    View full-size slide

  11. Benefits of Offline First

    View full-size slide

  12. Faster User Experience
    Better, faster user experience — both offline and online

    View full-size slide

  13. Works Offline
    Ability to disconnect and continue to work offline

    View full-size slide

  14. Battery and Bandwidth
    Limited access to power and communications infrastructure in disaster scenarios

    View full-size slide

  15. Offline-First Patterns and
    Anti-Patterns

    View full-size slide

  16. Tools and Use Cases

    View full-size slide

  17. CouchDB Replication Protocol
    @BradleyHolt
    Cloudant Sync
    CouchDB PouchDB
    CouchDB Replication Protocol
    CouchDB

    View full-size slide

  18. @BradleyHolt

    View full-size slide

  19. PouchDB Code Examples
    github.com/bradley-holt/offline-first

    View full-size slide

  20. JSON Documents
    {

    _id: "6EF9D2B0-13D3-1378-8D30-39E3CE0B36C2",

    _rev: "1-0b457efcf82fb29492ef927ba5b6ee15",

    type: "Feature",

    geometry: {

    type: "Point",

    coordinates: [

    -71.1028,

    42.3691

    ]

    },

    properties: {

    session_id: "3486b13f-7b8a-8a96-dfbf-9b82800e367f",

    timestamp: 1422928591717

    }

    }
    @BradleyHolt

    View full-size slide

  21. Creating a PouchDB Database
    var db = new PouchDB("smart-meter");
    @BradleyHolt

    View full-size slide

  22. Creating a New Document
    var db = new PouchDB("smart-meter");
    db.put({
    _id: "2014-11-12T23:27:03.794Z",
    kilowatt_hours: 14
    }).then(function() {
    console.log("Document created");
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt

    View full-size slide

  23. Updating a Document
    db.put({
    _id: "2014-11-12T23:27:03.794Z",
    kilowatt_hours: 14
    }).then(function(response) {
    return db.get(response.id);
    }).then(function(doc) {
    // Update the value for kilowatt hours
    doc.kilowatt_hours = 15;
    // Put the document back to the database
    return db.put(doc);
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt

    View full-size slide

  24. Querying a Database with allDocs
    db.bulkDocs([
    {_id: "2014-11-12T23:27:03.794Z", kilowatt_hours: 14},
    {_id: "2014-11-13T00:52:01.471Z", kilowatt_hours: 15},
    {_id: "2014-11-13T01:39:28.911Z", kilowatt_hours: 16},
    {_id: "2014-11-13T02:52:01.471Z", kilowatt_hours: 17}
    ]).then(function(result) {
    // Get all documents
    return db.allDocs({include_docs: true});
    }).then(function(response) {
    console.log(response);
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt

    View full-size slide

  25. allDocs Options
    §  include_docs
    – conflicts
    – attachments
    §  startkey
    §  endkey
    §  inclusive_end
    (true by default)
    §  limit
    §  skip
    §  descending
    §  key
    §  keys
    @BradleyHolt

    View full-size slide

  26. Querying a Database with Map/Reduce
    §  Most queries can be done with allDocs (in PouchDB)
    §  Map functions transform documents into indexes
    §  Reduce functions aggregate results of Map functions
    – _sum
    – _count
    – _stats
    @BradleyHolt

    View full-size slide

  27. Querying a Database with PouchDB Find
    §  Based on Cloudant Query (Mango)
    §  Declarative indexes
    §  MongoDB-style query language
    @BradleyHolt

    View full-size slide

  28. PouchDB Data Storage Limits
    @BradleyHolt
    Firefox Chrome Opera 15+ Internet
    Exporer
    10+
    iOS Safari Safari
    (desktop)
    Android PhoneGap /
    Cordova
    Data
    Storage
    Limit
    50MB
    (more with
    user
    permission)
    calculated calculated
    250MB
    (prompts
    user at 10
    MB)
    50MB
    (prompts
    user at 5MB
    and at
    increments)
    unlimited
    (prompts
    user at 5MB
    and at
    increments)
    calculated /
    200MB
    unlimited
    Adapter IndexedDB
    IndexedDB
    / WebSQL
    IndexedDB
    / WebSQL
    IndexedDB WebSQL WebSQL
    IndexedDB /
    WebSQL
    SQLite

    View full-size slide

  29. Apache CouchDB
    CouchDB is a document database featuring an HTTP API, JSON documents,
    and peer-to-peer replication
    @BradleyHolt

    View full-size slide

  30. Creating a Remote PouchDB Database
    var remoteDb = new PouchDB("https://bradley-holt.cloudant.com/smart-meter");
    @BradleyHolt

    View full-size slide

  31. Cross-Origin Resource Sharing (CORS)
    A security restriction implemented by browsers on cross-site HTTP requests
    @BradleyHolt

    View full-size slide

  32. Bidirectional Replication
    @BradleyHolt

    View full-size slide

  33. Bidirectional Replication
    db.sync(remoteDb, {
    live: false,
    retry: false
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    }).on("error", function(error) {
    // Replication has stopped due to an unrecoverable failure
    console.log(error);
    });
    @BradleyHolt

    View full-size slide

  34. Live Replication
    @BradleyHolt

    View full-size slide

  35. Live Replication
    var sync = db.sync(remoteDb, {
    live: true,
    retry: true
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    }).on("error", function(error) {
    // Replication has stopped due to an unrecoverable failure
    console.log(error);
    });
    @BradleyHolt

    View full-size slide

  36. Filtered Replication
    @BradleyHolt

    View full-size slide

  37. Filtered Replication
    db.replicate.to(remoteDb, {
    filter: function(doc) {
    return doc._id >= "2014-11-13T00:00:00.000Z";
    }
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    });
    @BradleyHolt

    View full-size slide

  38. One Database Per User
    @BradleyHolt
    userdb-d76846
    userdb-905cec
    userdb-adc95b
    userdb-c082f2
    userdb-730bba
    userdb-c3d3e5
    userdb-a1ec1f
    userdb-85a327
    userdb-9b9aba
    userdb-85bcfe
    rdb-da3d25

    View full-size slide

  39. Boilerplates & Tools
    §  Frontend Web Apps
    –  React Boilerplate with Service Workers

    §  Backend Web Apps
    –  PouchDB npm Package

    –  PouchDB Server npm Package

    §  Mobile Apps
    –  PouchDB for Ionic Framework

    –  "Hello world" Cordova app with PouchDB

    –  "Hello world" Cordova app with PouchDB, using the SQLite Plugin

    –  Cloudant FoodTracker (uses Cloudant Sync for iOS)

    §  Desktop Apps
    –  PouchDB for Electron (formerly Atom Shell)

    –  PouchDB for Chrome packaged apps

    –  "Hello world" Chrome app with PouchDB

    –  PouchDB for NW.js (aka Node-Webkit)

    §  Internet of Things (IoT) Apps
    –  Node-RED

    @BradleyHolt

    View full-size slide

  40. Cloudant FoodTracker
    An offline-first demo app built with Swift and Cloudant Sync for iOS

    View full-size slide

  41. Offline Camp
    June 24th - 26th, Catskill Mountains

    View full-size slide

  42. offlinefirst.org/camp

    View full-size slide

  43. Image Credits
    §  Open for Data Dome (outside) by Bradley Holt
    §  Open for Data Dome (inside) by Bradley Holt
    §  Brent Spiner (Data from Star Trek: The Next Generation) with
    Zoltar from Big by Bradley Holt, on Twitter

    §  A mockup of the golden Apple iPhone 5S by Zach Vega,
    on Wikimedia Commons

    §  Joan Touzet (@wohali), ASF Member, CouchDB PMC Member

    §  Device landscape by Jeremy Keith, on Flickr
    §  Cloud Formation Over the Adirondacks by Bradley Holt, on Twitter

    §  Cell phone tower by Gary Lerude, on Flickr
    §  Pneumatic Central by Sleestak, on Flickr
    §  Colunas by Daniel Zanini H., on Flickr
    §  Speed DLR on Doklands by Umberto Rotundo, on Flickr

    §  Waterfall by Paulo Valdivieso, on Flickr
    §  Wildfire by U.S. Fish and Wildlife Service Southeast Region, on Flickr

    §  Arduino Uno by Pete Prodoehl, on Flickr
    §  Mango with section on a white background by bangdoll, on Flickr

    §  Warning by Stefano Brivio, on Flickr
    @BradleyHolt

    View full-size slide

  44. Questions?
    @BradleyHolt

    View full-size slide