Visualization with Blaze and Bokeh

Visualization with Blaze and Bokeh

9bb4a79e5379e3d6705fd99a229d76ee?s=128

Andy R. Terrel

October 15, 2014
Tweet

Transcript

  1. Visualizing Data with Blaze and Bokeh Presented by: Andy R.

    Terrel & Peter Wang
  2. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 About Continuum Analytics Intro Large scale data analytics Interactive data visualization A practical example http://continuum.io/ ! ! We build technologies that enable analysts and data scientist to answer questions from the data all around us. Committed to Open Source Areas of Focus • Software solutions • Consulting • Training • Anaconda: Free Python distribution • Numba, Conda, Blaze, Bokeh, dynd • Sponsor
  3. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Visualizing Twitter Data with Blaze and Bokeh, Christine Doig, PyTexas 2014 About Andy Andy R. Terrel @aterrel Chief Scientist, Continuum Analytics ! President, NumFOCUS ! Background: • High Performance Computing • Computational Mathematics • President, NumFOCUS foundation ! Experience analyzing diverse datasets: • Finance • Simulations • Web data • Social media !
  4. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Visualizing Twitter Data with Blaze and Bokeh, Christine Doig, PyTexas 2014 About Peter Peter Wang @pwang Co-Founder, President Continuum Analytics ! Background: • Graphics • Scientific computing ! Experience: ! • Creator of Chaco and Bokeh • Founder of Streamitive ! ! !
  5. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 About this talk Visualizing Data with Blaze and Bokeh 1. Large scale data analytics - Blaze ! 2. Interactive data visualization - Bokeh ! 3. Example Intro Large scale data analytics Interactive data visualization A practical example Introduction to large-scale data analytics and interactive visualization through a tweets dataset practical example Objective Structure
  6. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example “ At my company X, we have peta/terabytes of data, just lying around, waiting for someone to explore it” - someone at PyTexas Let’s make it easier for users to explore and extract useful insights out of data. Package manager Free enterprise-ready Python distribution Anaconda Conda Blaze Bokeh Numba Wakari Power to speed up Share and deploy Interactive data visualizations Scale
  7. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 7 Blaze
  8. • Dealing with data applications has numerous pain points
 -

    Hundreds of data formats - Basic programs expect all data to fit in memory - Data analysis pipelines constantly changing from one form to another - Sharing analysis contains significant overhead to configure systems - Parallelizing analysis requires expert in particular distributed computing stack Data Pain
  9. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Large scale data analytics - An Overview BI - DB DM/Stats/ML Scientific Computing Distributed Systems Numba bcolz RHadoop
  10. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Blaze Source: http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign/
  11. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Distributed Systems Scientific Computing BI - DB DM/Stats/ML Blaze bcolz Connecting technologies to users Connecting technologies to each other Blaze hdf5
  12. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Data Storage Abstract expressions Computational backend csv HDF5 bcolz DataFrame Intro Large scale data analytics Interactive data visualization A practical example HDFS selection filter group by join column wise Pandas Streaming Python Spark MongoDB SQLAlchemy json Blaze
  13. Deferred Expr Compilers Interpreters Data Compute API Blaze Architecture •

    Flexible architecture to accommodate exploration
 • Use compilation of deferred expressions to optimize data interactions
  14. Deferred Expr Blaze Expr temps.hdf5 nasdaq.sql tweets.json Join by date

    Select NYC Find Tech Selloff Plot • Lazy computation to minimize data movement
 • Simple DAG for
 compilation to • parallel application • distributed memory • static optimizations
  15. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Data Storage Abstract expressions Computational backend csv HDF5 bcolz DataFrame HDFS selection filter group by join column wise Pandas Streaming Python Spark MongoDB SQLAlchemy json Blaze.expressions
  16. Blaze Data • Single interface for data layers
 • Composition

    of different
 formats
 • Simple api to add 
 custom data formats SQL CSV HDFS JSON Mem Custom HDF5 Data
  17. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Data Storage Abstract expressions Computational backend csv HDF5 bcolz DataFrame HDFS selection filter group by join column wise Pandas Streaming Python Spark MongoDB SQLAlchemy json Blaze.data
  18. Blaze Compute Compute DyND Pandas PyTables Spark • Computation abstraction

    over numerous data libraries
 • Simple multi-dispatched visitors to implement new backends
 • Allows plumbing between stacks to be seamless to user
  19. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Data Storage Abstract expressions Computational backend csv HDF5 bcolz DataFrame HDFS selection filter group by join column wise Pandas Streaming Python Spark MongoDB SQLAlchemy json Blaze.compute
  20. Blaze Example - Counting Weblinks Common Blaze Code #  Expr

      t_idx  =  TableSymbol('{name:  string,                                              node_id:  int32}')   t_arc  =  TableSymbol('{node_out:  int32,                                              node_id:  int32}')   joined  =  Join(t_arc,  t_idx,  "node_id")   t  =  By(joined,  joined['name'],                  joined['node_id'].count())   ! #  Data  Load   idx,  arc  =  load_data()
 #  Computations   ans  =  compute(t,  {t_arc:  arc,  t_idx:  idx})
 in_deg  =  dict(ans)   in_deg[u'blogspot.com']
  21. Blaze Example - Counting Weblinks Using Spark + HDFS load_data

    sc  =  SparkContext("local",  "Simple  App")   idx  =  sc.textFile(“hdfs://master.continuum.io/example_index.txt”)   idx  =  idx.map(lambda  x:  x.split(‘\t’))\                    .map(lambda  x:  [x[0],  int(x[1])])   arc  =  sc.textFile("hdfs://master.continuum.io/example_arcs.txt")   arc  =  arc.map(lambda  x:  x.split(‘\t’))\                    .map(lambda  x:  [int(x[0]),  int(x[1])])   Using Pandas + Local Disc with  open("example_index.txt")  as  f:          idx  =  [  ln.strip().split('\t')  for  ln  in  f.readlines()]   idx  =  DataFrame(idx,  columns=['name',  'node_id'])   ! with  open("example_arcs.txt")  as  f:          arc  =  [  ln.strip().split('\t')  for  ln  in  f.readlines()]   arc  =  DataFrame(arc,  columns=['node_out',  'node_id'])
  22. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Blaze.API Table Using the interactive Table object we can interact with a variety of computational backends with the familiarity of a local DataFrame
  23. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 23 Intro Large scale data analytics Interactive data visualization A practical example Blaze.API Table
  24. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Blaze.API Migrations - into the into function makes it easy to moves data from one container type to another
  25. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 25 Blaze notebooks Intro Large scale data analytics Interactive data visualization A practical example
  26. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Why I like using Blaze? ! - Syntax is very similar to Pandas - Easy to scale - Easy to find best computational backend to a particular dataset - Easy to adapt my code if someone handles me a dataset in a different format/ backend
  27. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Want to learn more about Blaze? Free Webinar: http://www.continuum.io/webinars/getting-started-with-blaze ! Blogpost: http://continuum.io/blog/blaze-expressions http://continuum.io/blog/blaze-migrations http://continuum.io/blog/blaze-hmda ! Docs and source code: http://blaze.pydata.org/ https://github.com/ContinuumIO/blaze
  28. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Data visualization - An Overview Results presentation Visual analytics Static Interactive Small datasets Large datasets Traditional plots Novel graphics
  29. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Bokeh • Interactive visualization • Novel graphics • Streaming, dynamic, large data • For the browser, with or without a server • Matplotlib compatibility • No need to write Javascript http://bokeh.pydata.org/ https://github.com/ContinuumIO/bokeh
  30. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Intro Large scale data analytics Interactive data visualization A practical example Bokeh - Interactive, Visual analytics • Tools (e.g. Pan, Wheel Zoom, Save, Resize, Select, Reset View)
  31. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 31 Intro Large scale data analytics Interactive data visualization A practical example Bokeh - Interactive, Visual analytics • Widgets and dashboards
  32. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 32 Bokeh - Interactive, Visual analytics Intro Large scale data analytics Interactive data visualization A practical example • Crossfilter
  33. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 33 Bokeh - Large datasets Server-side downsampling and abstract rendering Intro Large scale data analytics Interactive data visualization A practical example
  34. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 34 Bokeh - No JavaScript Intro Large scale data analytics Interactive data visualization A practical example
  35. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 35 Bokeh examples Intro Large scale data analytics Interactive data visualization A practical example
  36. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 A practical example: Twitter dataset Intro Large scale data analytics Interactive data visualization A practical example
  37. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Setting up your environment ! 1. Download Anaconda https://store.continuum.io/cshop/anaconda/ 2. Create your conda environment $conda create -n strata python=2.7 blaze bokeh ! 3. Activate environment $source activate strata $conda install mongodb pyspark ! 4. Start notebook $ipython notebook ! 5. Execute the notebooks Intro Large scale data analytics Interactive data visualization A practical example
  38. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Want to know more about Mining Twitter? Intro Large scale data analytics Interactive data visualization A practical example https://github.com/ptwobrussell/Mining-the-Social-Web-2nd-Edition/
  39. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 39 Twitter notebooks Intro Large scale data analytics Interactive data visualization A practical example
  40. Visualizing Data with Blaze and Bokeh, Strata / Hadoop World

    NYC 2014 Thank you! :)