MongoDB 3.4 - A Features Tour

At Netconomy's 3rd Innovation Day (13th Jan. 2017) I gave a talk about selected features in MongoDB 3.4. Here is a slightly modified/extended version of the slide deck of my session.

https://www.netconomy.net/

Transcript

1. MongoDB  3.4  Features   Hans-­‐Peter  Grahsl    @hpgrahsl       NETCONOMY  SoAware  &  ConsulDng  GmbH   3rd  InnovaDon  Day,  Fri  13th  Jan.  2017  
2. //    Outline                     NoSQL  Landscape  &  MongoDB   Main  New  Features  in  3.4   Live-­‐Demos  &  Samples  
3. //  NoSQL  Databases     “just  a  term”  referring  to  non-­‐relaDonal  database  systems     very  different  approaches  to  handle  data  persistence     not  a  single  agreed-­‐upon  definiDon     common  characterisDcs?     any  non-­‐relaDonal  data  model     mostly  a  very  good  fit  for  clustered  environments     designed  for  large/web-­‐scale  applicaDons   no  fixed  schema     typically  open-­‐source  (at  least  their  origins)    
4. //  4  NoSQL  Data  Models     Key  Values   Column  Family   Documents   Graph  
5. //  4  NoSQL  Data  Models     Key  Values   Column  Family   Graph  
6. //  Data  PlaYorms  Map  2016      
7. //  Data  PlaYorms  Map  2016      
8. //  MongoDB  Documents   Documents     { '_id' : 1, 'name' : { 'first' : 'John', 'last' : 'Backus' }, 'contribs' : [ 'Fortran', 'ALGOL', 'Backus-Naur Form', 'FP' ], 'awards' : [ { 'award' : 'W.W. McDowell Award', 'year' : 1967, 'by' : 'IEEE Computer Society' } ] }
9. //  MongoDB  Documents   Documents     rich  nested  JSON-­‐based/-­‐like  structure     internal  format:  BSON*   §  binary-­‐encoded  superset  of  JSON   §  more  datatypes   §  fast  encoding/decoding   §  efficient  in  storage  space   document  size  à  max.  16  MB     *details  see  h+p://bsonspec.org    
10. //  MongoDB       •  Databases   –  SQL  =  Database   •  CollecDons   –  SQL  ~  Table   •  Documents   –  SQL  ~  Rows     Namespaces:   db.collecDon  =>  iot.measurements   …  
11. //  MongoDB  CollecDons   CollecDons   don‘t  enforce  or  necessarily  exhibit  a  fixed  structure   structure  is  given  and  defined  on  document  level   may  contain  documents  of  arbitrary  structure     in  pracDce  documents  within  a  collecDon  are  mostly  similar  in  structure   vs.  
12. //  MongoDB  High-­‐Level     *   *  Be  careful  about  consistency:  there  are  several  op;ons.  In  fact  you  get  tunable  consistency     by  means  of  different  deployment  op;ons,  R/W  quorum  seCngs  etc.  which  in  turn  give  you   either  weaker  or  stronger  consistency  guarantees...  Choose  wisely!  
13. //  MongoDB  Persistence     Flexible  &  Pluggable  Storage  Engines     helps  to  address  various  needs  for  different  use  cases     MongoDB  Data  Model   MongoDB  Query  Language   WiredTiger   MMAPv1   In-­‐Memory   Encrypted   …   3rd  parDes  
14. //   Tooling   OperaDons   Development   What‘s  new  in  version  3.4?   main  new  features  in  three  categories  
15. //  Features  -­‐  Development   Views     non-­‐materialized  /  read-­‐only   provide  easy  data  access  (hide  aggregaDons)   redact  document  fields  (hide  sensiDve  data)   more  fine-­‐grained  security  (no  direct  access)   Tooling   OperaDons   Develop ment  
16. //  Features  -­‐  Development   Views     “syntheDc  collecDons“   don‘t  directly  contain  any  documents   built  based  on  aggregaDon  pipelines   view  definiDons  à  system.views  collecDon   no  writes  allowed  for  obvious  reasons   Tooling   OperaDons   Develop ment  
17. //  Features  -­‐  Development   Views  –  DefiniDon  /  CreaDon   creaDng  views  is  as  simple  as   db.createView( "view-name","src-collection“, <$agg_pipeline> ) Tooling   OperaDons   Develop ment  
18. //  Features  -­‐  Development   Views  –  Index  Usage?   views  have  no  aeached  indexes   however  can  use  indexes  of  underlying  collecDons   the  same  rules  apply  as  for  aggregaDons   Tooling   OperaDons   Develop ment  
19. //  Features  -­‐  Development   Views  –  Demo     Tooling   OperaDons   Develop ment  
20. //  Features  -­‐  Development   AggregaDon  Framework  Enhancements       $sample  (since  3.2)     $lookup  (since  3.2)     $graphLookup       $facet  /  $bucket  /  $bucketAuto     $addFields  /  $replaceRoot     $switch     and  a  bunch  of  array/string/ISOdate/...  operators   Tooling   OperaDons   Develop ment  
21. //  Features  -­‐  Development   AggregaDon  –  $graphLookup       used  for  analyzing  tree  or  graph  structures   logical  extension  of  $lookup  (eq.  lei-­‐outer  join)   works  by  doing  recursive  lookups   query  filters  and  depth  can  be  defined   supports  „self“  or  „cross“  collecDons  lookups   Tooling   OperaDons   Develop ment  
22. //  Features  -­‐  Development     e.g.  Hierarchy  of  Categories     Tooling   OperaDons   Develop ment   Mugs   Kitchen  &   Dining   Commuter  &   Travel   Glassware  &   Drinkware   Outdoor   RecreaDon   Camping   Mugs   Running   Thermos   Red  Run   Thermos   White  Run   Thermos   Blue  Run   Thermos   {  "_id"  :  1,  "name"  :  "Mugs"  }   {  "_id"  :  2,  "name"  :  "Kitchen  &  Dining",  "parentId"  :  1  }   {  "_id"  :  3,  "name"  :  "Commuter  &  Travel",  "parentId"  :  2  }   …   {  "_id"  :  10,  "name"  :  "Blue  Run  Thermos",  "parentId"  :  7  }  
23. //  Features  -­‐  Development     e.g.  Flight  Network     Tooling   OperaDons   Develop ment  
24. //  Features  -­‐  Development   $graphLookup:  {    from:  <collecDon>,    startWith:  <expression>,    connectFromField:  <string>,    connectToField:  <string>,    as:  <array>,    maxDepth:  <number>,    depthField:  <string>,    restrictSearchWithMatch:  <document>  }       Tooling   OperaDons   Develop ment  
25. //  Features  -­‐  Development   $graphLookup  –  Demo     Tooling   OperaDons   Develop ment  
26. //  Features  -­‐  Development   AggregaDon  –  $facet  /  $bucket(Auto)       manipulate,  inspect  &  analyze  mulD-­‐dimension  data   perform  faceted  search  &  navigaDon   facets  as  filters  to  narrow  result  sets   use  cases:  catalog  browsing  &  grouping  analyDcs,...     Tooling   OperaDons   Develop ment  
27. //  Features  -­‐  Development   e.g.  Catalog  Browsing  with  Facets   Tooling   OperaDons   Develop ment  
28. //  Features  -­‐  Development   AggregaDon  –  $bucket   allows  us  to  explicitly  define  range  boundaries   grouping  is  then  done  for  each  bucket   we  can  specify  a  default  bucket   we  can  reshape  the  resultset  structure   Tooling   OperaDons   Develop ment   boundaries:   0   40   80   150   1000   Other   buckets:   1   2   3   4   5   6   ranges:   0-­‐39   40-­‐79   80-­‐149   150-­‐999   1000   rest  
29. //  Features  -­‐  Development   AggregaDon  –  $bucket   db.collection.aggregate([ { $bucket: { groupBy: <expression>, boundaries: […,…], default: <literal>, output: { <output1>: {$accum_expr}, … } }} ]) Tooling   OperaDons   Develop ment  
30. //  Features  -­‐  Development   $bucket  –  Demo     Tooling   OperaDons   Develop ment  
31. //  Features  -­‐  Development   AggregaDon  –  $bucketAuto   only  hinDng  on  #buckets  to  split  into   finds  boundaries  automagically     tries  to  evenly  distribute  data  across  buckets   output  reshaping  also  supported   granularity  can  be  specified  for  numeric  types   Tooling   OperaDons   Develop ment  
32. //  Features  -­‐  Development   AggregaDon  –  $bucketAuto   db.collection.aggregate([ { $bucketAuto: { groupBy: <expression>, buckets: <number>, granularity: <string>, output: { <output1>: {$accum_expr}, … } }} ]) Tooling   OperaDons   Develop ment  
33. //  Features  -­‐  Development   $bucketAuto  –  Demo     Tooling   OperaDons   Develop ment  
34. //  Features  -­‐  Development   AggregaDon  –  $facet  à  „bringing  it  all  together...“   db.collection.aggregate([ { $facet: { <output_1>: [<$stage_1,…,$stage_n], … <output_n>: [<$stage_1,…,$stage_n], }} ]) Tooling   OperaDons   Develop ment  
35. //  Features  -­‐  Development   Demo  combinaDon  of  $facet/$bucket(Auto)     Tooling   OperaDons   Develop ment  
36. //  Features  -­‐  Development   other  notable  features  including     tunable  consistency  (linearizable  reads  J  finally)     extensive  collaDons  support  (several  languages)   new  BSON  Type  Decimal128  (exact  precision/ rounding)     a  bunch  of  new  Array/String/ISOdate  operators     ...   Tooling   OperaDons   Develop ment  
37. //  Features  -­‐  Tooling     Compass  1.5.1   new  SQL/BI  Connector   new  Apache  Spark  Connector   Tooling   OperaDons   Develop ment  
38. //  Features  -­‐  Tooling   Compass  –  new  cross  plaYorm  UI     current  features:   schema  visualizaDon  (incl.  geospaDal  data)   full  CRUD  capabiliDes   visual  explain  plans   index  management  &  usage  stats   rules  creaDon  for  document  validaDon   basic  real-­‐Dme  performance  metrics   Tooling   OperaDons   Develop ment  
39. //  Features  -­‐  Tooling  
40. //  Features  -­‐  Tooling   BI  Connector  2.0   completely  re-­‐implemented   access  MongoDB  data  with  favorite  tools     auto-­‐schema  sampling  &  generaDon   more  push-­‐down  to  database   supports  authenDcaDon  
41. //  Features  -­‐  OperaDons   ReplicaDon  Enhancements   beter  iniDal  sync  (Primary  à  Secondaries)     all  indexes  (not  only  _id)  are  being  build  during  the  sync   retry  logic  to  compensate  network  issues   intra  cluster  compression   snappy  to  achieve  up  to  70%  compression   benefits  w.r.t.  network  throughput   reduce  network  traffic  costs  in  cloud  environments   Tooling   OperaDons   Develop ment  
42. //  Features  -­‐  OperaDons   Sharding  Enhancements   full  membership  awareness  for  cluster  components   faster  balancing  due  to  parallel  chunk  migraDons   balancer  runs  on  config  servers  primary  node   config  servers  run  as  normal  replica  sets  (no  mirroring)   sharding  zones  (supersede  old  tag-­‐ware  sharding)   Tooling   OperaDons   Develop ment  
43. //  Great  tooling  for  (Dev)Ops  
44. //  Great  tooling  for  (Dev)Ops   MongoDB  Cloud  Manager   automaDon   monitoring   backup   cluster  mgmt  
45. //   ?   ?   ?   Thanks  for   your  interest!   QuesDons