Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using AWS to build ELK stack

ocowchun
September 20, 2017

Using AWS to build ELK stack

分享如何使用 AWS 來建置 ELK stack

ocowchun

September 20, 2017
Tweet

More Decks by ocowchun

Other Decks in Programming

Transcript

  1. AWS Elasticsearch 猻讨 — ਻ฃ藲碉襑ᥝጱ秚瑊碍, 犋襑ᥝᛔ૩蒂ቘ cluster ፘ橕ጱ戔ਧ膏 load balancer

    — 獉ୌ Kibana — ᛔ㵕ୌ缏猋犩 耬讨 — 斃膑ጱ粚๜(5.5, ፓ獮 Elasticsearch 磧碝傶 5.6.1) — EBS ጱ褖ګ, ྯ㮆 Instance Type 磪 EBS ጱӤ褖 (t2.small 傶 35 GB) — 櫞አጱ authenicate — 墋ᴱጱ羬翄虻რ碍硁
  2. ဳ఺ shard ጱ碍ᰁ shard ฎӞ㮆 Lucene instance 牧揗揣ୌ缏 document 膏䁆ᤈ

    query牐毆戔Ӟ㮆 index 䨝磪Բ㮆 shards, 吚 Elasticsearch ጱ index 虋ग़碻牧 shard 䨝蚤茐ीے牧襑ᥝ懿䛂誢犖䨝蚤茐ीے牧䌘ෝ logging 㬵藯蝢ଉ䨝಩ shard 戔ੜӞ讨牧仂੝羬翄ጱ揗砾牐 PUT https://elk.codementor.io/_template/logstash1 { "order": 1, "template": "logstash-*", "settings": { "number_of_shards" : 1 } }
  3. AWS Elasticsearch Access Policy — AWS Elasticsearch ጱ authenicate ݝر战獋圵憒㳷

    IAM 膏 IP — 䌘ෝ襑ᥝٌ犢 Authenicate 秚ګጱ䛑አ㬵藯盄犋ො׎牐
  4. Identity and Access Management (IAM) — Securely control access to

    AWS services and resources — IAM 磪 User 膏 Role 獋圵觊獨 — IAM User 蝢ଉฎ妔樄咳ᘏ౲ฎक़蟂ጱ๐率ֵአ(i.e. Heroku) — IAM Role 蝢ଉฎአ㬵瞲窔粬ਧ AWS resources ጱ稗褖 (i.e. ر战 EC2 ݢ犥蝫矑 S3)
  5. Proxy — ֵአ Proxy 㬵ୌ缏ᛔ૩ጱ Authenicate 秚ګ — 疥 Proxy

    Instance ጱ IAM Role ے獈 AWS Elasticsearch ጱ access policy — 礬硁襑穩牧戔ਧ Proxy 䌘क़ጱ Authenicate 秚ګ (i.e. HTTP auth, JWT ...)
  6. Logstash Authenicate http plugin 磪൉׀ http auth ጱ戔ਧݢ犥ֵአ input {

    http { port => 5000 user => "cm-logstash" password => "${LOGSTASH_PASSWORD}" } }
  7. Logstash Queue — By default, Logstash uses in-memory bounded queues

    between pipeline stages (inputs → pipeline workers) to buffer events. — The size of these in-memory queues is fixed and not configurable. — Absorbs bursts of events without needing an external buffering mechanism like Redis or Apache Kafka. queue.type: persisted queue.max_bytes: 1gb
  8. EC2 Container Service(ECS) — AWS ൉׀ጱ Container service 牧ඪൔ Docker

    — 䒻֦蒂ቘ Cluster ጱ Operation(ਞ蕕, 緳矒, 硌疻)牧ݢ犥斕 誫ጱ Scale out/down — ٌ䋿疰ฎ蕕ԧ ECS Container Agent ጱ EC2
  9. 膏 EC2 穉斃ጱ猻讨 — Docker instance ጱ珸㵕蝧ଶ穉 VM 盠 (seconds

    v.s minutes) — 犋አᛔ૩ᓕ Docker Cluster ጱ Operation — ֵአ Dockerfile 㬵ୌᗝ Image 牧ፘ䌘盠蝧膏墋㻌 — Ӟݣ秚瑊አ犋ਠጱ羬翄虻რݢ犥አ㬵狶ٌ犢Ԫ眐
  10. ECS — Task: 磧ੜ螀֢㻌֖ጱፘ橕戔ਧ牧۱珀Ӟکग़奲ጱ Container Definitions牧䌘䛑ጱ IAM Role牧禊盢Ӥ觊犲 Docker Compose

    file — Container Definitions: Container ጱፘ橕戔ਧ牧۱珀 Container ጱ image ֖ᗝ牏䌘क़樄硯ጱ Port牏絑ह虋碍牏襑ᥝጱ羬翄虻რ (CPU, Memory) — Cluster: 䋿褬䁆ᤈ Container ጱ瑿ො牧ྯ㮆 Cluster ݢ犥珸አग़㮆 Task — Repositories: AWS ൉׀ጱ Docker Image Repositories, ݢ犥疥 ୌᗝঅጱ Image 硯ک蝡愊
  11. Cluster: — Ӟ㮆 Cluster ݢ犥磪ग़㮆 services 膏 tasks牧犋螂蝢ଉ傶ԧො׎ᓕ ቘ牧౯㮉䨝螡䢔ֵአ service

    — ྯ㮆 Service ᮷襑ᥝ瞲ਧӞ犩 Task Definition 牧ݚक़螭襑ᥝ瞲ਧ䁆ᤈ ጱ碍ፓ膏翕᪠戔ਧ — 翕᪠戔ਧԆᥝฎ虏 service ݢ犥䌘क़ᶎ蝫矑牧क़ᶎݢ犥ฎ Internet ౲ฎ Intranet — 戔ਧঅԏ盅牧 Service 䨝ᛔ㵕䒻֦蒂ቘ珸አ Container牧 Health Check, Task Placement ጱ㵕֢牐 — 戔ਧ Minimum healthy percent, Maximum percent 㬵究ਧๅ碝 Services 碻ጱᤈ傶
  12. Application Load Balancer (ALB) 螕ݳ Container ጱ Load Balancer —

    ݢ犥礬硁 Host, path 㬵瞲ਧ䌘䛑ጱ service — ඪൔ WebSocket, HTTP/2 — ݢ犥蚤 ECS 狶碉ݳ
  13. 蟂ᗟ窕纷 — 礬硁襑穩ࣁ樄咳絑हୌᗝ Dokcerfile ҁi.e. 䌃Ӟ㮆 Logstash ጱ Dockerfile҂ —

    Ӥ㯽ک AWS ECS Repositories牧㪔秂ဳ䌘䛑ጱ tag — ๅ碝 Container 䌘䛑ጱ Task, ֵአ碝ጱ tag — ๅ碝 Service 䌘䛑ጱ Task Version — ECS Cluster 䨝ᛔ㵕礬硁 Service ጱ戔ਧ㬵珸አ碝ጱ Container 膏橕樂膑ጱ Container
  14. ECS ጱ㺔氂 — ᓕቘ Container ጱ log — ᓕቘ Container

    ጱ environment variable — ᓕቘ Container ጱ虻რֵአ眐丆 — 墋玕蟂ᗟጱ窕纷
  15. ref — Deploying the ELK stack on AWS ECS, Part

    1: Introduction & First Steps — The Hitchhiker's Guide to AWS ECS and Docker — AWSome ECS