Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Where Big Data and Semantic Intersect

September 22, 2014

Where Big Data and Semantic Intersect

Big Data, Semantics and the interesting opportunities at the intersection


September 22, 2014

More Decks by Valentin

Other Decks in Technology


  1. About  this  talk   •  Goal:  (Hopefully)  leave  you  with

     a  few   new  ideas  on  interes;ng  opportuni;es   at  the  intersec;on  of  Big  Data  and   Seman;cs   •  Structure     – Big  Data  &  Analy;cs   – Seman;c  Technologies   – The  intersec;on   – Three  examples  /  opportuni;es  
  2. About  Me   •  13  year  experience  in  soIware  driven

      innova;on   – seman;c  technologies  researcher  in  the  group  of   Prof  Studer   – manager  of  a  research  division  concerned  with  all   aspects  of  informa;on  driven  decisions  at  FZI   – Big  Data  consultant  with  codecen;c     – (analy;cs  consultant  with  Daimler  TSS)      
  3. Data  Deluge:  Moore’s  Law,  Mobile  Web,  Cloud   Compu;ng,  Data

     Value  Chain,  Social  Web  etc.  make   collec;ng  large  datasets  ever  cheaper      
  4. Big  Data  Technology:  lower  the  cost  to  build  systems  

    that  do  more  complex  processing  with  more  data   faster  
  5. (Big  Data)  Analy0cs:  build  on  this  data  (technologies)   to

     harness  paUerns  in  data  to  maximize  business   value  
  6. Example:  GPS,  Telemetrics,  handhelds,  mobile  web   etc.  drive  data

     deluge  in  postal  services  but  …    
  7. ..  This  must  be  harnessed  to  realize  same  day  

    delivery,  preven;ve  maintenance,  precise  arrival   es;mates  …  in  order  to  stay  compe;;ve    
  8. BeUer  informa;on  processing  through  more   considera;on  for  the  explicit

     context  of  processed   elements   Is  label  for   Is  label  for   has  capital   has  popula;on   in  country   contains   Lebanon   Lebanon,   Country   Beirut,   City   4  Million   USA   Dartmouth   Medical  S.   Lebanon,     NH  
  9. Seman;c  Technologies  by  Task   •  Discovering  Context:  understand  the

     meaning   of  unstructured  data  (text,  images,  …)   •  Moving  Context:  transfer  meaning  of  data   between  systems  (RDF,  RuleML,  …)   •  Using  Context:  inference  based  on  the   meaning  (inference  engines,  deduc;ve   databases)  
  10. Seman;c   Big  Data   ?   BigData  Technologies  for

     Seman0cs,  e.g.  using  a   hadoop  cluster  for  rule  inferencing  –  not  the  topic   today  
  11. Seman;c   Big  Data   ?   Seman0cs  for  Big

     Data,  using  seman;c  technologies   to  make  processing  large  amounts  of  data  simpler  
  12. Big  Picture  of  Big  Data  Systems   Ingest   Stage

      Transform   Serve   Cluster  Management   DFS  /  DDBMS   Resource  Management   Data  Flow  Management  
  13. e.g.     Ka_a,  Sqoop  &  Oozie   HBase  

    MR,    Pig  &  Storm   Hive  +  Cassandra   Ambari  +  ZooKeeper   HDFS   YARN   Apache  Falcon  
  14. (some)  seman;cs  relevant  challenges   in  Big  Data  systems  

    Ingest   Stage   Transform   Serve   Integra;ng  data   from  diverse   sources   Understanding   unstructured  /   polystructured   data   Languages  to   specify   transforma;ons  
  15. Moving  Context  with   JSON-­‐LD   Ingest   Stage  

    Transform   Serve   Integra;ng  data   from  diverse   sources  
  16. Observa0on:  Cudng  edge  enterprise  soIware   architecture  bears  a  striking

     resemblance  to  the  web   Source:  Mar;n  Fowler   Microservices,  REST,  ROCA,  Polyglot  Persistence  
  17. JSON  as  in  JavaScript  Object  Nota0on???   {! "firstName": "John",!

    "lastName": "Smith”,"age": 25,! "phoneNumbers": [! {! "type": "home",! "number": "212 555-1234"! },! {! "type": "office",! "number": "646 555-4567"! }! ],! "children": [],!
  18. Through  simplicity,  the  prolifera;on  of  JavaScript  and   through  a

     good  fit  to  other  data  structures  JSON  has   become  the  standard  for  data  interchange  on  the  web  
  19. JSON-­‐LD  allows  to  add  some  seman0cs  to  JSON   {!

    "@context": {! "name": "http://xmlns.com/foaf/0.1/name",! "homepage": {! "@id": "http://xmlns.com/foaf/0.1/workplaceHomepage",! "@type": "@id"! },! "Person": "http://xmlns.com/foaf/0.1/Person"! },! "@id": "http://me.markus-lanthaler.com",! "@type": "Person",! "name": "Markus Lanthaler",! "homepage": "http://www.tugraz.at/"! }!
  20. I  believe  the  linked  data  techniques  that  worked   for

     web-­‐scale  data  integra;on  can  offer  long   term  relief  for  the  Enterprise  data  integra;on   challenge  (and  that  JSON-­‐LD  can  help  in  doing   this)  
  21. Discovering  Seman0cs   in  the  Data  Lake   Ingest  

    Stage   Transform   Serve   Understanding   unstructured  /   polystructured   data  
  22. Mo;vated  by  the  need  for  agility  in  data  use  and

     the   availability  of  tools  to  cheaply  manage  giant  amounts  of   polystructured  data  enterprises  are  moving  from  a   tradi;onal  ETL-­‐Data  Warehouse  architecture  …     …  
  23. However,  there  is  currently  a  giant  gap  between   capabili;es

     of  companies  to  directly  u;lize  this  heap   of  polystructured  data  …  (e.g.  Elas;c  Search  +  Kibana    
  24. and  what  has  been  demonstrated  to  be  possible  with  

    such  heaps  of  polystructured  data  (e.g.  Cogni;ve   Compu;ng  and  IBMs  Watson  or  …  
  25. or  Probabilis;c  knowledge  fusion  and  Google   Knowledge  Vault  

    Dong,  Xin  Luna,  K.  Murphy,  E.  Gabrilovich,  G.   Heitz,  W.  Horn,  N.  Lao,  Thomas  Strohmann,   Shaohua  Sun,  and  Wei  Zhang.  "Knowledge   Vault:  A  Web-­‐scale  approach  to  probabilis;c   knowledge  fusion."  (2014).  
  26. I  believe  some  next  genera;on  Big  Data  leaders   will

     bring  Seman;cs  (as  in  “discovering  and   using  the  meaning  of  heaps  of  polystructured   data”)  to  many  more  enterprises  
  27. LP  for  View  Defini0ons   Ingest   Stage   Transform

      Serve   Languages  to   specify   transforma;ons   (e.g.  Cascalog)  
  28. Ingest   Stage   Transform   Serve   Linked  

    Enterprise  Data   (with  JSON-­‐LD)   connect  /  download  slides  at   www.vzach.de   Seman;cs  in  the   Data  Lake   LP  for  view   defini;ons     (e.g.  Cascalog)