Elastic{ON} 2018 - Upgrade to 6.0: Leading with Empathy

Slide 1

Slide 1 text

archana sriram kibana engineer elastic eldnahevitaerc upgrade to 6.0 leading with empathy archanid ^ change

Slide 2

Slide 2 text

who am I

Slide 3

Slide 3 text

黄花きばな

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

who is Elastic

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

who are Elastic

Slide 8

Slide 8 text

Elasticsearch: Core/Infra, Distributed, Search & Aggs, Lucene, Security Documentation Kibana Operations Kibana Platform Monitoring (Kibana & Elasticsearch) Beats Logstash Cloud Support Kibana QA

Slide 9

Slide 9 text

who are you

Slide 10

Slide 10 text

www.elastic.c o

Slide 11

Slide 11 text

How to ruin your elasticsearch installation

Slide 12

Slide 12 text

How to ruin your elasticsearch installation A heart-rending tragicomedy

Slide 13

Slide 13 text

How to ruin your elasticsearch installation A heart-rending tragicomedy A recurring nightmare

Slide 14

Slide 14 text

How to ruin your elasticsearch installation A heart-rending tragicomedy A recurring nightmare A true story

Slide 15

Slide 15 text

5.6 5.6 5.6

Slide 16

Slide 16 text

5.6 5.6 5.6

Slide 17

Slide 17 text

5.6 5.6

Slide 18

Slide 18 text

5.6 5.6 6.2

Slide 19

Slide 19 text

5.6 5.6 6.2

Slide 20

Slide 20 text

[2018-01-27T01:06:02,635][INFO ][o.e.d.z.ZenDiscovery ] [chmgl9ap15] failed to send join request to master [{chmgl9ap16}{- enENLKCQnaYPJW7IL80YA}{sK8- LS1OQhe9zOCTXJpdFg}{10.212.81.133}{10.212.81.133:9300}{ml.ma x_open_jobs=10, ml.enabled=true}], reason [RemoteTransportException[[chmgl9ap16][10.212.81.133:9300][intern al:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[chmgl9ap15][10.212.81.53:9300][internal: discovery/zen/join/validate]]; nested: IllegalStateException[Security index is not on the current version [6] - The Upgrade API must be run for 6.x nodes to join the cluster]; ]

Slide 21

Slide 21 text

Slide 22

Slide 22 text

5.6 5.6 6.2 .security

Slide 23

Slide 23 text

5.6 5.6 .security

Slide 24

Slide 24 text

5.6 5.6 5.6 .security

Slide 25

Slide 25 text

[2018-01-27T03:13:53,725][ERROR][o.e.g.GatewayMetaState ] [chmgl9ap15] failed to read local state, exiting… org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:176, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-176.st]

Slide 26

Slide 26 text

[2018-01-27T03:13:53,725][ERROR][o.e.g.GatewayMetaState ] [chmgl9ap15] failed to read local state, exiting... org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:176, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-176.st]

Slide 27

Slide 27 text

5.6 5.6 5.6 .security cluster state

Slide 28

Slide 28 text

5.6 5.6 5.6 .security cluster state

Slide 29

Slide 29 text

5.6 5.6 5.6 .security cluster state

Slide 30

Slide 30 text

5.6 5.6 .security cluster state

Slide 31

Slide 31 text

5.6 5.6 5.6 .security cluster state

Slide 32

Slide 32 text

5.6 5.6 .security cluster state 5.6

Slide 33

Slide 33 text

5.6 5.6 .security cluster state 5.6

Slide 34

Slide 34 text

5.6 5.6 .security cluster state 6.2

Slide 35

Slide 35 text

.security cluster state 6.2 6.2 5.6

Slide 36

Slide 36 text

.security cluster state 6.2 6.2 6.2

Slide 37

Slide 37 text

bin/x-pack/users useradd yasmeen -p ***** -r superuser

Slide 38

Slide 38 text

• First, congratulations on using your literacy skills right now. People who read instructions are rare. • 30% of users read. 15% both read and understand what they read. • You have no idea how much time went into all this documentation. • I bet you will not click the link to the documentation in the previous bullet point. • Also, those statistics about users are made up. • The next bullet point is really not worth reading, I promise. Just skip over it. • Are you frustrated because you cannot both read this slide and hear what I’m saying? • If I said nonsense right now, would you even notice? How to upgrade without borking your system 38 All the steps you need to follow to do it right! Here are the steps:

Slide 39

Slide 39 text

the 3 empathies

Slide 40

Slide 40 text

how it feels to use the product today

Slide 41

Slide 41 text

what’s best for the product

Slide 42

Slide 42 text

how it feels in the community as we change

Slide 43

Slide 43 text

structure flexibility

Slide 44

Slide 44 text

www.elastic.c o

Slide 45

Slide 45 text

innovate replace legacy features remove fragility improve user experience keep our users onboard Development Constitution

Slide 46

Slide 46 text

agility stability

Slide 47

Slide 47 text

upgrade assistant

Slide 48

Slide 48 text

upgrade guide

Slide 49

Slide 49 text

upgrade webinar

Slide 50

Slide 50 text

breaking changes docs

Slide 51

Slide 51 text

solutions for removal of multiple mapping types

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Kibana’s solution for removal of multiple mapping types

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

convert multiple types automatically*

Slide 57

Slide 57 text

www.elastic.c o

Slide 58

Slide 58 text

response-ability

Slide 59

Slide 59 text

{ }

Slide 60

Slide 60 text

www.elastic.c o

Slide 61

Slide 61 text

www.elastic.c o

Slide 62

Slide 62 text

the future

Slide 63

Slide 63 text

www.elastic.c o

Slide 64

Slide 64 text

Questions, Stories, Hugs, Tears Visit me at the AMA

Slide 65

Slide 65 text

www.elastic.c o

Slide 66

Slide 66 text

Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 66 Please attribute Elastic with a link to elastic.co

Slide 67

Slide 67 text

Appendix

Slide 68

Slide 68 text

Mapping types Analogy to SQL gone awry

Slide 69

Slide 69 text

user_name name email content tweeted_at user tweet

Slide 70

Slide 70 text

user_name name email content tweeted_at deleted user tweet

Slide 71

Slide 71 text

user_name name email content tweeted_at deleted user boolean tweet date

Slide 72

Slide 72 text

user_name name email content tweeted_at deleted user boolean tweet date

Slide 73

Slide 73 text

Split types into new indices

Slide 74

Slide 74 text

user_name name email content tweeted_at user tweet

Slide 75

Slide 75 text

user_name name email content tweeted_at user tweet user_name name email user user_name content tweeted_at tweet

Slide 76

Slide 76 text

PUT twitter { "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } }, "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } } }

Slide 77

Slide 77 text

PUT users { "settings": { "index.mapping.single_type": true }, "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } } } } PUT twitter { "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } }, "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } } }

Slide 78

Slide 78 text

PUT tweets { "settings": { "index.mapping.single_type": true }, "mappings": { "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } } } PUT users { "settings": { "index.mapping.single_type": true }, "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } } } } PUT twitter { "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } }, "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } } }

Slide 79

Slide 79 text

POST _reindex { "source": { "index": "twitter", "type": "tweet" }, "dest": { "index": "tweets" } } POST _reindex { "source": { "index": "twitter", "type": "user" }, "dest": { "index": "users" } }

Slide 80

Slide 80 text

GET users/doc/5 { "_index": "users", "_type": "doc", "_id": "5", "_version": 1, "found": true, "_source": { "user_name": "kimchy", "name": "Shay Banon", "email": "[email protected]" } } GET twitter/user/5 { "_index": "twitter", "_type": "user", "_id": "5", "_version": 1, "found": true, "_source": { "user_name": "kimchy", "name": "Shay Banon", "email": "[email protected]" } } OLD WAY TO GET A DOCUMENT NEW WAY TO GET A DOCUMENT

Slide 81

Slide 81 text

Slide 82

Slide 82 text

POST users/doc/5 { "user_name": "kimchy", "name": "Shay Banon", "email": "[email protected]" } POST twitter/user/5 { "user_name": "kimchy", "name": "Shay Banon", "email": "[email protected]" } OLD WAY TO POST A DOCUMENT NEW WAY TO POST A DOCUMENT

Slide 83

Slide 83 text

Merge fields into single type

Slide 84

Slide 84 text

user_name name email content tweeted_at user tweet

Slide 85

Slide 85 text

user_name name email content tweeted_at user tweet user_name name email content tweeted_at doc

Slide 86

Slide 86 text

user_name name email content tweeted_at user tweet type user_name name email content tweeted_at doc

Slide 87

Slide 87 text

user_name name email content tweeted_at user tweet type user_name name email content tweeted_at doc user tweet

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

PUT new_twitter { "settings": { "index.mapping.single_type": true }, "mappings": { "doc": { "properties": { "type": { "type": "keyword" }, "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" }, "content": { "type": "text" }, "tweeted_at": { "type": "date" } } } } } PUT twitter { "mappings": { "user": { "properties": { "name": { "type": "text" }, "user_name": { "type": "keyword" }, "email": { "type": "keyword" } } }, "tweet": { "properties": { "content": { "type": "text" }, "user_name": { "type": "keyword" }, "tweeted_at": { "type": "date" } } } } }

Slide 90

Slide 90 text

POST _reindex { "source": { "index": "twitter", }, "dest": { "index": "new_twitter" }, "script": { "source": """ ctx._source.type = ctx._type; ctx._id = ctx._type + '-' + ctx._id; ctx._type = 'doc'; """ } }

Slide 91

Slide 91 text

GET twitter/user/5 { "_index": "twitter", "_type": "user", "_id": "5", "_version": 1, "found": true, "_source": { "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } } GET new_twitter/doc/user-5 { "_index": "new_twitter", "_type": "doc", "_id": "user-5", "_version": 2, "found": true, "_source": { “type”: “user”, "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } } OLD WAY TO GET A DOCUMENT NEW WAY TO GET A DOCUMENT

Slide 92

Slide 92 text

GET new_twitter/doc/user-5 { "_index": "new_twitter", "_type": "doc", "_id": "user-5", "_version": 2, "found": true, "_source": { “type”: “user”, "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } } GET twitter/user/5 { "_index": "twitter", "_type": "user", "_id": "5", "_version": 1, "found": true, "_source": { "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } } OLD WAY TO GET A DOCUMENT NEW WAY TO GET A DOCUMENT

Slide 93

Slide 93 text

Slide 94

Slide 94 text

Slide 95

Slide 95 text

POST new_twitter/doc/user-5 { "type": "user", "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } POST twitter/user/5 { "user_name": "eldnahevitaerc", "name": "Archana Sriram", "email": "[email protected]" } OLD WAY TO POST A DOCUMENT NEW WAY TO POST A DOCUMENT

Slide 96

Slide 96 text

Nest multiple types under single top-level type

Slide 97

Slide 97 text

user_name name email content tweeted_at user tweet

Slide 98

Slide 98 text

user tweet doc type user_name name email type user_name content tweeted_at user_name name email content tweeted_at user tweet