Content-as-a-Service

This talk is centred around building almost real time news search engine with fine details about scalable and fast responsive ( average 10 ms) system architecture.

Transcript

1. Content as a Service ”Building real time news search engine” Heeren Sharma Software Engineer Cliqz GmbH
2. Cliqz • ~3 Million+ downloads across all platforms and products • Search Engine : ~2 Billion Indexed Pages (We don’t believe in indexing the web) • Privacy to User : Anti-Tracking & Anti- Phishing • Open Source Contributions • News recommendations
3. Cliqz
4. Cliqz
5. Line of thoughts • Power of Content • System Architecture • News Search • Challenges
6. Content is everywhere
7. Power of News Information Hub Keep up with events Shapes our reality
8. System Architecture - Overview • Streaming Data • Python, Redis and Elasticsearch (core technologies) • Full Containerized – Micro-service architecture • Scalable – Horizontal and Vertical • Quick Responsiveness - ~10ms (avg)
9. System Architecture – Data Sources & Pipelines
10. System Architecture – Indexing & API
11. System Architecture – News-Indexer Indexer Aliasmanager Updater Web API
12. Cliqz News Technology - Search • Custom Query Search Language • Real time ranking • Topical Clustering • Trending News Events
13. Cliqz News Technology – Search - QSL ”Tom Petty” site:spiegel.de url_path:musik contrib_twt_ids:2834511 lang:de
14. Cliqz News Technology – Search - Real-time Ranking • Cliqz Boost Algorithm • Function Scores (Elasticsearch) • Mix of Signals
15. Cliqz News Technology – Search - Real-time Ranking: Function Scores • Query Part • Script Score Part doc["engagement"].value*36000 / (params.now - doc["firstseen_at"].value)
16. Cliqz News Technology – Search - Clustering and Trends • Cluster articles over same topic – Unsupervised • Representing terms from each cluster • Permutations and combinations – Word(s) transformations
17. Self Serving System • Semi-Automated Data Source update • Expressive search queries • Embed your own ranking algorithms • Easy Data Insights – aggregations, dashboards, etc. • Extensive Experience in newsletters curation and editing
18. Bumps in Journey #challenges #lessons-learned • Data modelling • Performance takes extra mile • Test Early • Think Scalable Architecture • Reliability of sources
19. Great Powers – Greater responsibility
20. Thank you for your attention! Heeren Sharma @heerensharma heerensharma@yahoo.com heeren@cliqz.com Madalina Diana Sandu Team Lead - News mds@cliqz.com Cliqz It!
21. Fake News