Elasticsearch in Anger: Stories from the GitHub Search Clusters

This talk was presented at the inaugural Elastic{ON} conference, http://elasticon.com

Session Abstract:

Over the past two years GitHub’s source code search product has grown from a small research project into a very large index containing nearly 4 billion documents. This is an ever changing and continuously growing data set that has presented us with some interesting scaling problems. This talk will cover how they have tackled these scaling problems - from monitoring and alerting, application changes, growing clusters, and tuning Lucene parameters.

Presented by Tim Pease, GitHub

Transcript

1. Elasticsearch in Anger Tim Pease stories from the GitHub search clusters "
2. " ‣Where have we come from ‣What we are doing now ‣Where we are going # $ %
4. Code Search Load
5. “you can find secret keys people have committed to their public repositories”
6. Code Search Load
8. Drew Raines
9. " ‣We performed inadequate load testing ‣We had insufficient operations experience ‣We need better tools and metrics # $ % Code Search
10. Code Search Query Performance
11. New Cluster
13. HAProxy for Multiple Clusters Application HAProxy 9210:githubsearch3 9211:codesearch1 9210 9211 githubsearch3 codesearch1
14. Push Button Index Creation
15. Create a New Index
16. Backfill Data
17. Index Versions & x <SHA1 Hexdigest> settings mappings (
18. mappings: { index-meta: { _meta: { version: <SHA1 Hexdigest> } } } Index Versions & x
19. Because …
20. GitHub Enterprise
21. Load Testing
22. Load Testing Scientist github.com/github/scientist ‣control ‣experiment ‣throttling
23. Load Testing require "scientist" def search(query) science "code-search-load-test" do |e| e.use { old_index.search(query) } e.try { new_index.search(query) } end end
24. " ‣We were outgrowing our old cluster ‣We created migration tools ‣We used production queries for load testing # $ % New Cluster
25. New Cluster
26. New Cluster New Queries
27. Elasticsearch Training
28. { query: {constant_score: { filter: { term: {state: “open"} } }} } { query: {match_all: {}}, filter: { term: {state: “open"} } } Tale of Two Queries
29. Query Facets Filter Filter Query Facets Filter * Filter
30. author:grantr assignee:kimchi mentions:TwP involves:drewr label:bug repo:elasticsearch/elasticsearch created:<2015-01-01 updated:>2015-03-01 is:issue is:open Common Issue Filters
31. author:grantr assignee:kimchi mentions:TwP involves:drewr label:bug repo:elasticsearch/elasticsearch created:<2015-01-01 updated:>2015-03-01 is:issue is:open Common Issue Filters
33. " ‣We were able to look at query performance ‣We got some education about filters ‣We now enjoy efficient filtered queries # $ % New Queries
34. New Cluster New Queries
35. New Cluster New Queries Heap Exhaustion
36. Code Search Heap Usage
37. Code Search Heap Usage Out Of Memory
40. node name | disk | used | free | percent ----------------------+------+------+------+-------- codesearch-storage1 6.9T 5.9T 1022G 86% codesearch-storage2 6.9T 6.2T 699G 91% codesearch-storage3 6.9T 6.1T 841G 89% codesearch-storage4 6.9T 6.0T 935G 87% codesearch-storage5 6.9T 6.3T 630G 92% codesearch-storage6 6.9T 6.2T 672G 91% codesearch-storage7 6.9T 6.1T 859G 88% codesearch-storage8 6.9T 6.1T 843G 88% codesearch-storage9 6.9T 6.1T 870G 88% codesearch-storage10 6.9T 6.0T 921G 87%
41. Add Capacity
42. Logical Volume Manager Add Capacity
43. 7.6 TB 6.9 TB
44. 7.6 TB 7.6 TB
45. How Did This Happen?
46. Total Cache Size Lucene Segments JVM Heap Usage
47. Prevention
49. /es forecast disk codesearch1 codesearch1 will reach 70% disk usage in 302 days (2015-12-31) with 93% confidence
50. " ‣We were ignoring key metrics ‣We added alerts for key metrics ‣We created tools to forecast growth # $ % Heap Exhaustion
51. New Cluster New Queries Heap Exhaustion
52. New Cluster New Queries Heap Exhaustion New Cluster
53. One More Story
54. Upgrade to ES 1.4.2
55. Upgrade to ES 1.4.2
56. Upgrade to ES 1.4.2
57. Hot Threads 97.4% (487.1ms out of 500ms) cpu usage by thread 'elasticsearch[githubsearch3-storage1-cp1-prd][management][T#2]' 9/10 snapshots sharing following 9 elements org.elasticsearch.action.admin.indices.stats.ShardStats.<init>(ShardStats.java:49) 97.3% (486.3ms out of 500ms) cpu usage by thread 'elasticsearch[githubsearch3-storage1-cp1-prd][management][T#3]' 2/10 snapshots sharing following 20 elements java.io.UnixFileSystem.getLength(Native Method) 96.4% (482.1ms out of 500ms) cpu usage by thread 'elasticsearch[githubsearch3-storage1-cp1-prd][management][T#4]' 2/10 snapshots sharing following 19 elements org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:223)
58. Management Threads
59. Amen Sampler /_nodes/_local/stats
60. HAProxy Change /_nodes/_local/stats ➠ /
61. HAProxy Change
62. " ‣We were missing some important metrics ‣We have an entire ecosystem ‣We have confidence in ES 1.4.2 # $ % Upgrade to ES 1.4.2
63. " ‣Where have we come from ‣What we are doing now ‣Where we are going # $ %
64. Where have we come from # #
65. What are we doing now $ $
66. Where are we going /es forecast disk codesearch1 codesearch1 will reach 70% disk usage in 302 days (2015-12-31) with 93% confidence % %
67. " GitHub Search Team Grant Rodgers Tim Pease github.com/grantr github.com/TwP
68. " The End