Roblox Elasticsearch Meetup

Slides from Guru's presentation at the Elastic Silicon Valley meetup on 5/5/2016.

Transcript

1. May 6, 2016 Elasticsearch Meetup Guru
2. What is ROBLOX ROBLOX is the Imagination Platform™ Every month over 12 Million users and 30 Million visitors come to ROBLOX to play, adventure, imagine and create with their friends. • #1 gaming platform for Kids and Teens • Primarily user generated content • 360k concurrent players at peak times • 800K+ requests/sec
3. What is ROBLOX
4. How ROBLOX uses Elasticsearch Group Search Message Search Games Search ROBLOX Imagination Platform Catalog Search Leaderboards User Search Virtual Sales
5. What’s Under the Hood
6. What’s Under the Hood
7. • The ROBLOX Catalog contains items uploaded by the community and by us • Makes use of boosting, filtering, custom scripts, tokenizers, and analyzers • eCPM sort • 350 Million documents (11 shards) • Seven nodes User Generated Catalog
8. How ROBLOX uses Elasticsearch Group Search Message Search Games Search ROBLOX Imagination Platform Catalog Search Leaderboards User Search Virtual Sales
9. • Game developers can award as many points as they want to the players • Each game gets a leaderboard but all the per-game leaderboard documents go into the same index • Aggregate leaderboards viewable by day, week, month and all time Leaderboards
10. • Used Amazon DynamoDB to count the time-bucketed points per game • 200 Million documents (six shards) across six nodes • Watch out for the default query generated by your client Leaderboards (continued) Valid query 1: { "filter": { "term": { “gameId": "8888" } } } Valid query 2: { "query": { "filtered": { "filter": { "term": { " gameId": "8888" } } } } } took : 774 ms took : 53 ms
11. How ROBLOX uses Elasticsearch Group Search Message Search Games Search ROBLOX Imagination Platform Catalog Search Leaderboards User Search Virtual Sales
12. • 124 Million documents, 9 shards, 3 nodes • Partial word matches User Search
13. How ROBLOX uses Elasticsearch Group Search Message Search Games Search ROBLOX Imagination Platform Catalog Search Leaderboards User Search Virtual Sales
14. Query 1: { "and": { "filters": [ { "range": { "id": { "gt": "10" } } }, { "range": { "id": { "lt": "110" } } } ] } } • Gives the ability for our Customer Service team to respond to purchase anomalies and also ensure security • 800 Million documents (11 shards) across three nodes • 4 Million documents being added per day • Range query performance Users + Catalog = Virtual Sales Query 2: { "range": { "id": { "gt": "10", "lt": "110" } } } took: 5543 ms took:15 ms
15. • Migrated our servers along with elasticsearch clusters from Toronto to Chicago • Considered approaches like cross data-center replication of indices • Scheduled snapshots copied over by automated scripts • Snapshots restored in the new data center • Once the rest of the site migrated over, the data started flowing into the new indices Data Center Migration
16. ROBLOX + Elasticsearch Keeps the Imagination Platform Running Thank You