Share
Embed
Transcript
43 slides
-
Textalytics: Meaning-as-a-Service Real Time Semantic Search for Social TV streams
César de Pablo Sánchez Daedalus Nov. 7th 2013 Big Data Spain (Madrid) -
The plot 1. What's Social TV? 2. Monitoring Social TV
conversations. A preliminary architecture 3. Understanding the buzz. Textalytics 4. Organizing the mess. SenseiDB 5. Lessons learned -
API NLP and Semantics API Multilingual: EN, ES
(FR,IT,PT,CA) REST Service : JSON and XML Combine best of all worlds Deep language analysis Comprehensive resources: linguistics and Dbs Ontology Rule Based Method Statistics and Machine Learning Methods -
High level semantic API – close to bussines scenarios
Core API – building blocks Topics Sentiment Classif. Linked Data POS Configuración y Recursos Lingüísticos Configuración y Recursos Lingüísticos Configuración y Recursos Lingüísticos API Análisis Medios API Publicación Semántica … API -
Core API Topics Extraction Text Classification Sentiment Analysis Language identification
Lemmatization, POS and Parsing Speeech Recognition and Speaker Diarization Semantic Linked Data Viewer Semantic Linked Data Viewer Spell, Grammar and Style User Demographics -
Language identification Given a text identify a language list
- or just one 62 languages Using language ngrams signatures Social TV Filter – TV hashtags often implies language Sometimes hashtags are multilingual – but not relevant for users -
Text Classification Theme labels – IPTC Relevance
Multiple labels Tailored for short text (tweets) Define your own models and categories Social TV – filter on topic content -
Sentiment analysis Document level classification Positive/Negative/Neutral Subjective/Objective
Tailored for short texts Handles twitter jargon – RT, @, hashtags, emoticons, spelling errors, disfluence Other features Entity level sentiment Segment level sentiment -
Topics Extraction Personas: Ben Bernanke, Mariano Rajoy… Empresas,
Organizaciones: BBVA, Bankia, Goldman Sachs, Coca-Cola, Reserva Federal… Entidades económicas: Ibex35, Dax Xetra… Ubicaciones: Londres, EE.UU., París… Conceptos: prima de riesgo, presidente del Gobierno, intervención parlamentaria, índice bursátil, situación económica… Referencias de tiempo: hoy, ayer, sobre las 11 de la mañana… Cantidades económicas: 104 dólares, 1 euro… 12 main types Ontology with > 200 types Instances – BBVA Classes – bank fictional/historic SocialTV: populate custom -
Entity Linking Linking entities to their 'real' representation
Linking to several LOD sources -
SenseiDB Open source, distributed, realtime, semi- structured database
From LinkedIn sna: powering Linkedin home and LinkedIn signals Integrates other open source technologies: Zoie – search engine Bobo - faceted search Apache Kafkan – pub-sub system http://www.senseidb.com/ -
SenseiDB features 'Hybrid' Information Retrieval – Database Full
text search Structured and faceted search Fast real time updates with low latency and high troughput – pull model Single table/collection BQL – a SQL like language Eventual consistency Distributed – sharding and partitioning Hadoop integration -
Faceted search Amazon.com? Identify relevant attributes to use
as filters Predefined facets Defina a table schema Define fields as facets – facet schema -
Faceted search in depth Field types Basic: string,
int, short, long float, double, char Complex: date and text (analyzed, termvectors) Facet types Simple : 1 row – 1 value Hierarchical – Path c>b>a Range – define ranges Multi : 1 row – n values Histogram – define bins and their size TimeRange – for real time data -
Using facets for semantic search Define a facet for:
entities/concept → tweets about Chicote – include all variants + user + hashtags for each entity types → Navigate by type – Popular people classification/sentiment/emotions → Positive tweets about Chicote users or hashtags → popular users / popular mentions / correlated hashtags Urls Time range facets -
BQL – search, filter and facets Examples Facets
support basic analytics task defined as facets Relevance may be defined in query – text queries -
Real time indexing Data events – add and delete
Data streams – succession of data events Gateways Read data events from data streams File JDBC JMS Kafka Custom: Twitter -
Scalability Zookeper to keep replicas Low indexing latency
(no batch commit) Low search latency – low votality Horizontally scalable – shards Shards may be replicated N times Elastic – nodes can be added to accomodate growth -
Other features Batch indexing via Hadoop – ETL
Simple analytics by batch indexing Customized relevance models MapReduce functions over facets Sum, avg, min, max DistinctCount Activity values – volatile values – likes -
Conclusions SenseiDB is fast at searching/indexing – no variance
A couple nodes enough to handle Spanish SocialTV Love query language and time operators Support real time exploration -
Limitations SenseiDB Documentation is still scarce Single
table model – flat users and reputation Tricks to store complex facets Social TV Tracker Group entity mentions across tweets Relevance is tricky – ad hoc Manageability Integrate history -
Comparison Solr NearRT updates Soft commits
Simple facets Popular – great tools Storm, S4 ? ElasticSearch Batch commits On line facets Aggregation after facets Much better plugin system