Using AWS to build ELK stack

分享如何使用 AWS 來建置 ELK stack

Transcript

1. Using AWS to build ELK stack @ocowchun
2. ፓ秂 — 穉 logit 犵ጱ籧螀揲አ — Elasticsearch ݢ犥ො׎ጱ褰襑穩 scaling out/down, backup — Logstash ݢ犥ො׎ጱ褰襑穩 scaling out/down
3. AWS Elasticsearch 猻讨 — ਻ฃ藲碉襑ᥝጱ秚瑊碍, 犋襑ᥝᛔ૩蒂ቘ cluster ፘ橕ጱ戔ਧ膏 load balancer — 獉ୌ Kibana — ᛔ㵕ୌ缏猋犩 耬讨 — 斃膑ጱ粚๜(5.5, ፓ獮 Elasticsearch 磧碝傶 5.6.1) — EBS ጱ褖ګ, ྯ㮆 Instance Type 磪 EBS ጱӤ褖 (t2.small 傶 35 GB) — 櫞አጱ authenicate — 墋ᴱጱ羬翄虻რ碍硁
4. ဳ఺ shard ጱ碍ᰁ shard ฎӞ㮆 Lucene instance 牧揗揣ୌ缏 document 膏䁆ᤈ query牐毆戔Ӟ㮆 index 䨝磪Բ㮆 shards, 吚 Elasticsearch ጱ index 虋ग़碻牧 shard 䨝蚤茐ीے牧襑ᥝ懿䛂誢犖䨝蚤茐ीے牧䌘ෝ logging 㬵藯蝢ଉ䨝಩ shard 戔ੜӞ讨牧仂੝羬翄ጱ揗砾牐 PUT https://elk.codementor.io/_template/logstash1 { "order": 1, "template": "logstash-*", "settings": { "number_of_shards" : 1 } }
5. AWS Elasticsearch Access Policy — AWS Elasticsearch ጱ authenicate ݝر战獋圵憒㳷 IAM 膏 IP — 䌘ෝ襑ᥝٌ犢 Authenicate 秚ګጱ䛑አ㬵藯盄犋ො׎牐
6. Identity and Access Management (IAM) — Securely control access to AWS services and resources — IAM 磪 User 膏 Role 獋圵觊獨 — IAM User 蝢ଉฎ妔樄咳ᘏ౲ฎक़蟂ጱ๐率ֵአ(i.e. Heroku) — IAM Role 蝢ଉฎአ㬵瞲窔粬ਧ AWS resources ጱ稗褖 (i.e. ر战 EC2 ݢ犥蝫矑 S3)
7. Proxy — ֵአ Proxy 㬵ୌ缏ᛔ૩ጱ Authenicate 秚ګ — 疥 Proxy Instance ጱ IAM Role ے獈 AWS Elasticsearch ጱ access policy — 礬硁襑穩牧戔ਧ Proxy 䌘क़ጱ Authenicate 秚ګ (i.e. HTTP auth, JWT ...)
8. Logstash — ࢩ傶䷱磪制眲牧ݝฎ揗揣矑硩虻碘牧旉矦膏㯽蝑虻碘牐 — ݢ犥犨௔ጱ䌘盃犢牧穉ො藯憽஑羬翄磪㺔氂疰಩秚瑊橕ധ牧 樄Ӟݣ碝ጱ牐 — ࢩ傶ฎ Public ጱ๐率牧襑ᥝ Authenicate 秚ګ牐
9. Logstash Authenicate http plugin 磪൉׀ http auth ጱ戔ਧݢ犥ֵአ input { http { port => 5000 user => "cm-logstash" password => "${LOGSTASH_PASSWORD}" } }
10. Logstash Queue — By default, Logstash uses in-memory bounded queues between pipeline stages (inputs → pipeline workers) to buffer events. — The size of these in-memory queues is fixed and not configurable. — Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka. queue.type: persisted queue.max_bytes: 1gb
11. EC2 Container Service(ECS) — AWS ൉׀ጱ Container service 牧ඪൔ Docker — 䒻֦蒂ቘ Cluster ጱ Operation(ਞ蕕, 緳矒, 硌疻)牧ݢ犥斕 誫ጱ Scale out/down — ٌ䋿疰ฎ蕕ԧ ECS Container Agent ጱ EC2
12. 膏 EC2 穉斃ጱ猻讨 — Docker instance ጱ珸㵕蝧ଶ穉 VM 盠 (seconds v.s minutes) — 犋አᛔ૩ᓕ Docker Cluster ጱ Operation — ֵአ Dockerfile 㬵ୌᗝ Image 牧ፘ䌘盠蝧膏墋㻌 — Ӟݣ秚瑊አ犋ਠጱ羬翄虻რݢ犥አ㬵狶ٌ犢Ԫ眐
13. ECS — Task: 磧ੜ螀֢㻌֖ጱፘ橕戔ਧ牧۱珀Ӟکग़奲ጱ Container Definitions牧䌘䛑ጱ IAM Role牧禊盢Ӥ觊犲 Docker Compose file — Container Definitions: Container ጱፘ橕戔ਧ牧۱珀 Container ጱ image ֖ᗝ牏䌘क़樄硯ጱ Port牏絑ह虋碍牏襑ᥝጱ羬翄虻რ (CPU, Memory) — Cluster: 䋿褬䁆ᤈ Container ጱ瑿ො牧ྯ㮆 Cluster ݢ犥珸አग़㮆 Task — Repositories: AWS ൉׀ጱ Docker Image Repositories, ݢ犥疥 ୌᗝঅጱ Image 硯ک蝡愊
14. Cluster: — Ӟ㮆 Cluster ݢ犥磪ग़㮆 services 膏 tasks牧犋螂蝢ଉ傶ԧො׎ᓕ ቘ牧౯㮉䨝螡䢔ֵአ service — ྯ㮆 Service ᮷襑ᥝ瞲ਧӞ犩 Task Definition 牧ݚक़螭襑ᥝ瞲ਧ䁆ᤈ ጱ碍ፓ膏翕᪠戔ਧ — 翕᪠戔ਧԆᥝฎ虏 service ݢ犥䌘क़ᶎ蝫矑牧क़ᶎݢ犥ฎ Internet ౲ฎ Intranet — 戔ਧঅԏ盅牧 Service 䨝ᛔ㵕䒻֦蒂ቘ珸አ Container牧 Health Check, Task Placement ጱ㵕֢牐 — 戔ਧ Minimum healthy percent, Maximum percent 㬵究ਧๅ碝 Services 碻ጱᤈ傶
15. Application Load Balancer (ALB) 螕ݳ Container ጱ Load Balancer — ݢ犥礬硁 Host, path 㬵瞲ਧ䌘䛑ጱ service — ඪൔ WebSocket, HTTP/2 — ݢ犥蚤 ECS 狶碉ݳ
16. 蟂ᗟ窕纷 — 礬硁襑穩ࣁ樄咳絑हୌᗝ Dokcerfile ҁi.e. 䌃Ӟ㮆 Logstash ጱ Dockerfile҂ — Ӥ㯽ک AWS ECS Repositories牧㪔秂ဳ䌘䛑ጱ tag — ๅ碝 Container 䌘䛑ጱ Task, ֵአ碝ጱ tag — ๅ碝 Service 䌘䛑ጱ Task Version — ECS Cluster 䨝ᛔ㵕礬硁 Service ጱ戔ਧ㬵珸አ碝ጱ Container 膏橕樂膑ጱ Container
17. ECS ጱ㺔氂 — ᓕቘ Container ጱ log — ᓕቘ Container ጱ environment variable — ᓕቘ Container ጱ虻რֵአ眐丆 — 墋玕蟂ᗟጱ窕纷
19. ref — Deploying the ELK stack on AWS ECS, Part 1: Introduction & First Steps — The Hitchhiker's Guide to AWS ECS and Docker — AWSome ECS