From Big Data to Mobile Data with Apache CouchDB and PouchDB at Apache: Big Data

It’s all too easy for mobile app developers to assume that their apps will run on fast and reliable networks. The reality for end users, though, is often slow, unreliable networks with spotty coverage. What happens when the network doesn’t work, or when a device is in airplane mode? You get unhappy, frustrated users. One solution is to take an offline-first approach. An offline-first app is an app that works, without error, when there is no network connection. Offline-first apps built with Apache CouchDB and PouchDB (an open source JavaScript database) can provide better, faster user experiences by storing data locally and then synchronizing with a cloud database when a network connection is available.

Transcript

1. Apache: Big Data
   Bradley Holt, Developer Advocate
   Tuesday, May 10, 2016
   From Big Data to Mobile Data with
   Apache CouchDB and PouchDB
   @BradleyHolt
   

   View Slide

2. @BradleyHolt
   

   View Slide

3. View Slide

4. IBM Cloud Data Services
   Open for Data
   A comprehensive por.olio of open source data services
   

   View Slide

5. View Slide

6. View Slide

7. Big Data
   Get it?
   

   View Slide

8. Why offline first?
   

   View Slide

9. View Slide

10. View Slide

11. Mobile First
    Design for the smallest device first and then apply progressive enhancement techniques to
    take advantage of larger screen sizes when available
    

    View Slide

12. Offline First
    Design for offline usage first and then apply progressive enhancement techniques to take
    advantage of network connectivity when available
    

    View Slide

13. Ubiquitous Connectivity
    Why offline first in a world of ubiquitous connectivity?
    

    View Slide

14. The Eight Fallacies of Distributed Computing
    1.  The network is reliable
    2.  Latency is zero
    3.  Bandwidth is infinite
    4.  The network is secure
    5.  Topology doesn't change
    6.  There is one administrator
    7.  Transport cost is zero
    8.  The network is homogeneous
    @BradleyHolt
    

    View Slide

15. Mobile Backend
    What happens when your mobile backend service is unreachable?
    

    View Slide

16. Benefits of Offline First
    

    View Slide

17. Faster User Experience
    Better, faster user experience — both offline and online
    

    View Slide

18. Works Offline
    Ability to disconnect and continue to work offline
    

    View Slide

19. Battery and Bandwidth
    Limited access to power and communications infrastructure in disaster scenarios
    

    View Slide

20. Offline-First Patterns and
    Anti-Patterns
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. Tools and Use Cases
    

    View Slide

27. CouchDB Replication Protocol
    @BradleyHolt
    Cloudant Sync
    CouchDB PouchDB
    CouchDB Replication Protocol
    CouchDB
    

    View Slide

28. View Slide

29. @BradleyHolt
    

    View Slide

30. PouchDB Code Examples
    github.com/bradley-holt/offline-first
    

    View Slide

31. View Slide

32. JSON Documents
    {

    _id: "6EF9D2B0-13D3-1378-8D30-39E3CE0B36C2",

    _rev: "1-0b457efcf82fb29492ef927ba5b6ee15",

    type: "Feature",

    geometry: {

    type: "Point",

    coordinates: [

    -71.1028,

    42.3691

    ]

    },

    properties: {

    session_id: "3486b13f-7b8a-8a96-dfbf-9b82800e367f",

    timestamp: 1422928591717

    }

    }
    @BradleyHolt
    

    View Slide

33. Creating a PouchDB Database
    var db = new PouchDB("smart-meter");
    @BradleyHolt
    

    View Slide

34. Creating a New Document
    var db = new PouchDB("smart-meter");
    db.put({
    _id: "2014-11-12T23:27:03.794Z",
    kilowatt_hours: 14
    }).then(function() {
    console.log("Document created");
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt
    

    View Slide

35. Updating a Document
    db.put({
    _id: "2014-11-12T23:27:03.794Z",
    kilowatt_hours: 14
    }).then(function(response) {
    return db.get(response.id);
    }).then(function(doc) {
    // Update the value for kilowatt hours
    doc.kilowatt_hours = 15;
    // Put the document back to the database
    return db.put(doc);
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt
    

    View Slide

36. Querying a Database with allDocs
    db.bulkDocs([
    {_id: "2014-11-12T23:27:03.794Z", kilowatt_hours: 14},
    {_id: "2014-11-13T00:52:01.471Z", kilowatt_hours: 15},
    {_id: "2014-11-13T01:39:28.911Z", kilowatt_hours: 16},
    {_id: "2014-11-13T02:52:01.471Z", kilowatt_hours: 17}
    ]).then(function(result) {
    // Get all documents
    return db.allDocs({include_docs: true});
    }).then(function(response) {
    console.log(response);
    }).catch(function(error) {
    console.log(error);
    });
    @BradleyHolt
    

    View Slide

37. allDocs Options
    §  include_docs
    – conflicts
    – attachments
    §  startkey
    §  endkey
    §  inclusive_end
    (true by default)
    §  limit
    §  skip
    §  descending
    §  key
    §  keys
    @BradleyHolt
    

    View Slide

38. Querying a Database with Map/Reduce
    §  Most queries can be done with allDocs (in PouchDB)
    §  Map functions transform documents into indexes
    §  Reduce functions aggregate results of Map functions
    – _sum
    – _count
    – _stats
    @BradleyHolt
    

    View Slide

39. Querying a Database with PouchDB Find
    §  Based on Cloudant Query (Mango)
    §  Declarative indexes
    §  MongoDB-style query language
    @BradleyHolt
    

    View Slide

40. PouchDB Data Storage Limits
    @BradleyHolt
    Firefox Chrome Opera 15+ Internet
    Exporer
    10+
    iOS Safari Safari
    (desktop)
    Android PhoneGap /
    Cordova
    Data
    Storage
    Limit
    50MB
    (more with
    user
    permission)
    calculated calculated
    250MB
    (prompts
    user at 10
    MB)
    50MB
    (prompts
    user at 5MB
    and at
    increments)
    unlimited
    (prompts
    user at 5MB
    and at
    increments)
    calculated /
    200MB
    unlimited
    Adapter IndexedDB
    IndexedDB
    / WebSQL
    IndexedDB
    / WebSQL
    IndexedDB WebSQL WebSQL
    IndexedDB /
    WebSQL
    SQLite
    

    View Slide

41. Replication
    

    View Slide

42. Apache CouchDB
    CouchDB is a document database featuring an HTTP API, JSON documents,
    and peer-to-peer replication
    @BradleyHolt
    

    View Slide

43. View Slide

44. Creating a Remote PouchDB Database
    var remoteDb = new PouchDB("https://bradley-holt.cloudant.com/smart-meter");
    @BradleyHolt
    

    View Slide

45. Cross-Origin Resource Sharing (CORS)
    A security restriction implemented by browsers on cross-site HTTP requests
    @BradleyHolt
    

    View Slide

46. Bidirectional Replication
    @BradleyHolt
    

    View Slide

47. Bidirectional Replication
    db.sync(remoteDb, {
    live: false,
    retry: false
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    }).on("error", function(error) {
    // Replication has stopped due to an unrecoverable failure
    console.log(error);
    });
    @BradleyHolt
    

    View Slide

48. Live Replication
    @BradleyHolt
    

    View Slide

49. Live Replication
    var sync = db.sync(remoteDb, {
    live: true,
    retry: true
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    }).on("error", function(error) {
    // Replication has stopped due to an unrecoverable failure
    console.log(error);
    });
    @BradleyHolt
    

    View Slide

50. Filtered Replication
    @BradleyHolt
    

    View Slide

51. Filtered Replication
    db.replicate.to(remoteDb, {
    filter: function(doc) {
    return doc._id >= "2014-11-13T00:00:00.000Z";
    }
    }).on("change", function(info) {
    // Replication has written a new document
    console.log(info);
    }).on("complete", function(info) {
    // Replication has complete or been cancelled
    console.log(info);
    });
    @BradleyHolt
    

    View Slide

52. One Database Per User
    @BradleyHolt
    userdb-d76846
    userdb-905cec
    userdb-adc95b
    userdb-c082f2
    userdb-730bba
    userdb-c3d3e5
    userdb-a1ec1f
    userdb-85a327
    userdb-9b9aba
    userdb-85bcfe
    rdb-da3d25
    

    View Slide

53. View Slide

54. Boilerplates & Tools
    §  Frontend Web Apps
    –  React Boilerplate with Service Workers
    
    §  Backend Web Apps
    –  PouchDB npm Package
    
    –  PouchDB Server npm Package
    
    §  Mobile Apps
    –  PouchDB for Ionic Framework
    
    –  "Hello world" Cordova app with PouchDB
    
    –  "Hello world" Cordova app with PouchDB, using the SQLite Plugin
    
    –  Cloudant FoodTracker (uses Cloudant Sync for iOS)
    
    §  Desktop Apps
    –  PouchDB for Electron (formerly Atom Shell)
    
    –  PouchDB for Chrome packaged apps
    
    –  "Hello world" Chrome app with PouchDB
    
    –  PouchDB for NW.js (aka Node-Webkit)
    
    §  Internet of Things (IoT) Apps
    –  Node-RED
    
    @BradleyHolt
    

    View Slide

55. Cloudant FoodTracker
    An offline-first demo app built with Swift and Cloudant Sync for iOS
    

    View Slide

56. View Slide

57. Offline Camp
    June 24th - 26th, Catskill Mountains
    

    View Slide

58. offlinefirst.org/camp
    

    View Slide

59. Image Credits
    §  Open for Data Dome (outside) by Bradley Holt
    §  Open for Data Dome (inside) by Bradley Holt
    §  Brent Spiner (Data from Star Trek: The Next Generation) with
    Zoltar from Big by Bradley Holt, on Twitter
    
    §  A mockup of the golden Apple iPhone 5S by Zach Vega,
    on Wikimedia Commons
    
    §  Joan Touzet (@wohali), ASF Member, CouchDB PMC Member
    
    §  Device landscape by Jeremy Keith, on Flickr
    §  Cloud Formation Over the Adirondacks by Bradley Holt, on Twitter
    
    §  Cell phone tower by Gary Lerude, on Flickr
    §  Pneumatic Central by Sleestak, on Flickr
    §  Colunas by Daniel Zanini H., on Flickr
    §  Speed DLR on Doklands by Umberto Rotundo, on Flickr
    
    §  Waterfall by Paulo Valdivieso, on Flickr
    §  Wildfire by U.S. Fish and Wildlife Service Southeast Region, on Flickr
    
    §  Arduino Uno by Pete Prodoehl, on Flickr
    §  Mango with section on a white background by bangdoll, on Flickr
    
    §  Warning by Stefano Brivio, on Flickr
    @BradleyHolt
    

    View Slide

60. Questions?
    @BradleyHolt
    

    View Slide