Elasticsearch is a distributed, RESTful search engine "car" A car (or automobile) is a wheeled motor vehicle used for transportation. Most definitions of car say they run primarily on roads, seat one to eight people, have four tires, and mainly transport people rather than goods. Cars came into global use during the 20th century, and developed economies depend on them. The year 1886 is regarded as the birth year of the modern car, when German inventor Karl Benz built his Benz Patent-Motorwagen. Cars did not become widely available until the early 20th century. One of the first cars that was accessible to the masses was the 1908 Model T, an American car manufactured by the Ford Motor Company. Cars were rapidly adopted in the US, where they replaced animal-drawn carriages and carts, but took much longer to be accepted in Western Europe and other parts of the world.
M2X – datapoints Month New Datapoints July 2016 42,216,421 August 2016 38,456,296 September 2016 43,252,336 October 2016 89,572,942 November 2016 222,608,051 December 2016 338,588,651 January 2017 2,326,317,955 February 2017 7,192,489,182 ⠇ ⠇
#2 – Reduce number of indices you need to query … datapoints-20161229 datapoints-20161230 datapoints-20161231 datapoints-20170101 datapoints-20170102 datapoints-20170103 datapoints-20170104 datapoints-20170105 datapoints-20170106 datapoints-20170107 … … 29 December 2016 30 December 2016 31 December 2016 1 January 2017 2 January 2017 3 January 2017 4 January 2017 5 January 2017 6 January 2017 7 January 2017 … Device created
#2 – Reduce number of indices you need to query … datapoints-20161229 datapoints-20161230 datapoints-20161231 datapoints-20170101 datapoints-20170102 datapoints-20170103 datapoints-20170104 datapoints-20170105 datapoints-20170106 datapoints-20170107 … … 29 December 2016 30 December 2016 31 December 2016 1 January 2017 2 January 2017 3 January 2017 4 January 2017 5 January 2017 6 January 2017 7 January 2017 … Device created
#3 – Query less indices at once Q: Fetch 100 latest datapoints GET /all-datapoints/_search GET /v1-datapoints-20170616/_search GET /v1-datapoints-20170615/_search GET /v1-datapoints-20170614/_search GET /v1-datapoints-20170613/_search GET /v1-datapoints-20170612/_search …
#6 – Upgrade Elasticsearch v1.x Loads fields into memory and creates a data structure for searching at query time • Slow searches • No available memory for caching • OutOfMemory exceptions for large indices
#6 – Upgrade Elasticsearch v2.x Creates a columnar data structure on disk at write time • Fast searches • Small memory usage • Works for indices of any size
#7 – Scroll in large pages GET /datapoints/_search?size=1000&scroll=true GET /_search/_scroll GET /_search/_scroll GET /_search/_scroll GET /_search/_scroll …
#7 – Scroll in large pages GET /datapoints/_search?size=10000&scroll=true GET /_search/_scroll GET /_search/_scroll GET /_search/_scroll GET /_search/_scroll … 5x faster datapoint exports
#8 – Use cached counts when possible GET /_cat/indices index docs.count users 110843 … datapoints 43879824976 … devices 180301 streams 11793537 Datapoint count speedup from 18s to 0.7s
janko_m
prof18
little_rubyist
ymotongpoo
m_seki
rinchsan
mame
coe401_
kinocoboy2
ksudate
munetoshi
ffjlabo
pineapplecandy
sugarenia
quramy
chriscoyier
rasmusluckow
davidbonilla
holman
paigeccino
caitiem20
notwaldorf
brettharned
pedronauck