HipChat ElasticSearch meetup

"Zuhaib will discuss how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. He will cover some of the issues we've overcome using elasticsearch with AWS and CouchDB."

http://www.meetup.com/Elasticsearch-San-Francisco/events/141698772/

Transcript

1. Wednesday, November 20, 13
2. Heavy Lifting ZUHAIB SIDDIQUE • @ZUHAIB • ENGINEER Wednesday, November 20, 13
3. Wednesday, November 20, 13
4. 1. Persistent chat rooms Wednesday, November 20, 13
5. 2. File sharing & Chat history Wednesday, November 20, 13
6. 3. @mentions + notifications Wednesday, November 20, 13
7. 4. Integrations and API Wednesday, November 20, 13
8. 5. Runs on everything Wednesday, November 20, 13
9. 5. Runs on everything Wednesday, November 20, 13
10. 5. Runs on everything Wednesday, November 20, 13
11. Group chat and IM for teams. Wednesday, November 20, 13
12. 1,000,000,000 messages sent 1,200,000,000 Wednesday, November 20, 13
13. { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } } What is a HipChat Message? Wednesday, November 20, 13
14. In the beginning there was … Wednesday, November 20, 13
15. Then we added... https://github.com/rnewson/couchdb-lucene Wednesday, November 20, 13
16. CouchDB + CouchDB-Lucene •Started to tip over at 200M Docs •m2.4xlarge Wednesday, November 20, 13
17. CouchDB + CouchDB-Lucene m2.4xlarge m2.4xlarge Worked well for the first 500M Wednesday, November 20, 13
18. m2.4xlarge hs1.8xlarge Wednesday, November 20, 13
19. But it still wasn’t big enough. Wednesday, November 20, 13
20. Dont F*ck the Customer Wednesday, November 20, 13
21. First Attempt at Scaling Wednesday, November 20, 13
22. Wednesday, November 20, 13
23. Why ElasticSearch for HipChat Wednesday, November 20, 13
24. Why ElasticSearch for HipChat { "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" } } Wednesday, November 20, 13
25. Why ElasticSearch for HipChat Wednesday, November 20, 13
26. Search Error Rate Wednesday, November 20, 13
27. Wednesday, November 20, 13
28. Don’t Forget about CouchDB AWS Max EBS size Wednesday, November 20, 13
29. Stop the CouchDB bleeding Wednesday, November 20, 13
30. CouchDB end of the road Wednesday, November 20, 13
31. Wednesday, November 20, 13
32. (allthethings) ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } } ElasticSearch CouchDB Wednesday, November 20, 13
33. (allthethings) ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } } ElasticSearch CouchDB Wednesday, November 20, 13
34. CouchDB vs ElasticSearch CouchDB Spike to 800ms! Avg 100ms with ES Wednesday, November 20, 13
35. master Running ElasticSearch clients HipChat Customer Thanks Philip O’Toole - Loggly Wednesday, November 20, 13
36. $ fab create_esindex:hostname=esindex-d4 Wednesday, November 20, 13
37. •Spin up EC2 Instance •Install Oracle Java 7 •Create Provision IOPs EBS •Install and configure ElasticSearch •??? •Profit! Behind the Scenes Wednesday, November 20, 13
38. # Elasticsearch nodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] }) if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found" else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } } end Wednesday, November 20, 13
39. site.json "elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ] } Wednesday, November 20, 13
40. You Wednesday, November 20, 13
41. Running Elasticsearch Wednesday, November 20, 13
42. Running ElasticSearch Single Node OOM! Wednesday, November 20, 13
43. Monitor, Monitor, Monitor! Wednesday, November 20, 13
44. Backup Wednesday, November 20, 13
45. 100,000,000,000? To infinity and beyond! Wednesday, November 20, 13
46. Questions? Wednesday, November 20, 13