Terrafromで構築するマルチクラウドプラットフォームインフラストラクチャ / NDS53 Terraform

http://nagaoka.techtalk.jp/no53
https://github.com/hayajo/nds53-terraform

Transcript

1. 5FSSBGPSNͰߏங͢Δ
   ϚϧνΫϥ΢υϓϥοτϑΥʔϜ
   ΠϯϑϥετϥΫνϟ
   ௕Ԭ*5։ൃऀษڧձ /%4
   
   ୈճษڧձ
   
   

   View Slide

2. ࣗݾ঺հ
   w )BZBUP*NBJࠓҪ൏ਓ
   w !IBZBKP
   w Πϯϑϥ୲౰
   

   View Slide

3. 5FSSBGPSN
   

   View Slide

4. ΠϯϑϥΛ
   ϓϩϏδϣχϯά͢Δπʔϧ
   

   View Slide

5. ଟ͘ͷϓϥοτϑΥʔϜ΍
   αʔϏεΛαϙʔτ
   w *BB4
   "84 ($1 "[VSF 0QFO4UBDL
   w 1BB4
   )FSPLV
   w 4BB4
   'BTUMZ .BJM(VO /FX3FMJD
   w ଞʹ΋ͨ͘͞Μ
   w ֤ϓϥοτϑΥʔϜ΍αʔϏεݻ༗ͷଟ͘ͷϦιʔεʹରԠ
   

   View Slide

6. *OGSBTUSVDUVSFBT$PEF
   Λ࣮ݱ͢Δ
   w 5FSSBGPSN$POpHVSBUJPOϑΝΠϧʹΠϯϑϥ
   ߏ੒Λهड़
   w ෳ਺ͷϓϥοτϑΥʔϜ΍αʔϏεͰߏங͞Ε
   ΔΠϯϑϥετϥΫνϟશମΛදݱ
   w ϓϥοτϑΥʔϜ΍αʔϏε͝ͱʹಠཱͨ͠πʔ
   ϧΛ࢖͍Θ͚ͳͯ͘ྑ͍
   

   View Slide

7. 5FSSBGPSNʹΑΔ
   ΠϯϑϥߏஙͷྲྀΕ
   

   View Slide

8. ྫ&$&*1Λߏங͢Δ
   5FSSBGPSN$POpHVSBUJPO
   provider "aws" {
   region = "ap-northeast-1"
   }
   resource "aws_instance" "example" {
   ami = "ami-3bd3c45c"
   instance_type = "t2.micro"
   }
   resource "aws_eip" "example" {
   instance = "${aws_instance.example.id}"
   }
   IUUQTHJUIVCDPNIBZBKPOETUFSSBGPSN
   

   View Slide

9. ࡞ۀ؀ڥͷॳظԽ
   $ terraform init
   Initializing provider plugins...
   - Checking for available provider plugins on https://
   releases.hashicorp.com...
   - Downloading plugin for provider "aws" (1.0.0)...
   Terraform has been successfully initialized!
   

   View Slide

10. ࣮ߦܭըͷ֬ೝʢ̍ʣ
    $ terraform plan
    + aws_eip.example
    id:
    instance: "${aws_instance.example.id}"
    vpc:
    + aws_instance.example
    id:
    ami: "ami-3bd3c45c"
    instance_type: "t2.nano"
    subnet_id:
    Plan: 2 to add, 0 to change, 0 to destroy.
    

    View Slide

11. ࣮ߦܭըͷ֬ೝʢ̎ʣ
    $ terraform graph | dot -Tpng >tf.png && open tf.png
    

    View Slide

12. Πϯϑϥͷߏஙͱ֬ೝʢ̍ʣ
    $ terraform apply
    aws_instance.example: Creating...
    aws_instance.example: Creation complete after 16s (ID:
    i-0311a0b3adc495d2b)
    aws_eip.example: Creating...
    aws_eip.example: Creation complete after 1s (ID: eipalloc-
    a1c9a39b)
    Apply complete! Resources: 2 added, 0 changed, 0 destroyed
    

    View Slide

13. Πϯϑϥͷߏஙͱ֬ೝʢ̎ʣ
    $ terraform show
    aws_eip.example:
    id = eipalloc-a1c9a39b
    instance = i-0311a0b3adc495d2b
    public_ip = 13.115.163.200
    aws_instance.example:
    id = i-0311a0b3adc495d2b
    ami = ami-3bd3c45c
    instance_type = t2.nano
    w ঢ়ଶ͸UFSSBGPSNUGTUBUFϑΝΠϧͰ؅ཧ͞ΕΔ
    ϩʔΧϧʢσϑΥϧτʣ΍4ɺ($4ɺFUDEͳͲͷόοΫΤϯυͰ
    ؅ཧՄೳ
    όοΫΤϯυʹΑͬͯ͸ϩοΫ΋Մೳ
    

    View Slide

14. Πϯϑϥͷߋ৽
    $ terraform plan
    ~ aws_eip.example
    instance: "i-0311a0b3adc495d2b" => "$
    {aws_instance.example.id}"
    -/+ aws_instance.example (new resource required)
    id: "i-0311a0b3adc495d2b" =>
    (forces new resource)
    ami: "ami-3bd3c45c" => "ami-
    dfd0c7b8" (forces new resource)
    instance_type: "t2.nano" => "t2.nano"
    subnet_id: "subnet-ec29e9a5" =>
    Plan: 1 to add, 1 to change, 1 to destroy.
    ami = "ami-3bd3c45c" => "ami-dfd0c7b8"
    

    View Slide

15. Πϯϑϥͷഁغ
    $ terraform plan --destroy
    - aws_eip.example
    - aws_instance.example
    Plan: 0 to add, 0 to change, 2 to destroy.
    $ terraform destroy
    aws_eip.example: Destroying... (ID: eipalloc-a1c9a39b)
    aws_instance.example: Destroying... (ID:
    i-0311a0b3adc495d2b)
    Destroy complete! Resources: 2 destroyed.
    

    View Slide

16. 5FSSBGPSNͰߏங͢Δ
    ϚϧνΫϥ΢υϓϥοτϑΥʔϜ
    ΠϯϑϥετϥΫνϟ
    

    View Slide

17. ྫ,JOFTJTͱ#JH2VFSZͰ
    ߏங͢Δσʔλ෼ੳج൫
    IUUQTHJUIVCDPNIBZBKPOETUFSSBGPSN
    

    View Slide

18. ࡞ۀ؀ڥͷॳظԽ
    

    View Slide

19. terraform {
    required_version = "~> 0.10.0"
    }
    provider "aws" {
    access_key = "${var.aws["access_key"]}"
    secret_key = "${var.aws["secret_key"]}"
    region = "${var.aws["region"]}"
    }
    provider "google" {
    credentials = "${file(var.google["credentials"])}"
    project = "${var.google["project"]}"
    region = "${var.google["region"]}"
    }
    

    View Slide

20. variable "aws" {
    default = {
    access_key = ""
    secret_key = ""
    region = "ap-northeast-1"
    }
    }
    variable "google" {
    default = {
    credentials = "credentials.json"
    project = ""
    region = ""
    }
    }
    variable "service_name" {
    default = "nds53"
    }
    

    View Slide

21. ,JOFTJT4USFBN
    

    View Slide

22. resource "aws_kinesis_stream" "main" {
    name = "${var.service_name}-${terraform.workspace}"
    shard_count = 1
    }
    output "Kinesis stream" {
    value = "${aws_kinesis_stream.main.name}"
    }
    

    View Slide

23. #JH2VFSZ
    

    View Slide

24. resource "google_bigquery_dataset" "main" {
    # only [0-9a-zA-Z_]
    dataset_id = "${var.service_name}_${terraform.workspace}"
    }
    resource "google_bigquery_table" "main" {
    dataset_id = "${google_bigquery_dataset.main.dataset_id}"
    table_id = "${var.bq["table_id"]}"
    schema = "${file("bq/schema.json")}"
    }
    variable "bq" {
    default = {
    table_id = "myapp"
    }
    }
    output "BigQuery" {
    value = "${var.google["project"]}:$
    {google_bigquery_dataset.main.dataset_id}:$
    {google_bigquery_table.main.table_id}"
    }
    

    View Slide

25. [
    { "name": "time", "type": "timestamp", "mode": "required" },
    { "name": "tag", "type": "string", "mode": "required" },
    { "name": "value", "type": "float", "mode": "required" }
    ]
    CRTDIFNBKTPO
    

    View Slide

26. ,.4
    

    View Slide

27. resource "aws_kms_key" "main" {}
    resource "aws_kms_alias" "main" {
    name = "alias/${var.service_name}-${terraform.workspace}"
    target_key_id = "${aws_kms_key.main.key_id}"
    }
    

    View Slide

28. -BNCEB
    

    View Slide

29. data "archive_file" "kinesis2bq" {
    type = "zip"
    source_dir = "lambda/kinesis2bq"
    output_path = "lambda/kinesis2bq.zip"
    }
    resource "aws_lambda_function" "kinesis2bq" {
    filename = "${data.archive_file.kinesis2bq.output_path}"
    function_name = "${var.service_name}-${terraform.workspace}-kinesis2bq"
    role = "${aws_iam_role.kinesis_lambda.arn}"
    handler = "kinesis2bq.handler"
    source_code_hash = "${data.archive_file.kinesis2bq.output_base64sha256}"
    runtime = "python2.7"
    timeout = 60
    # NOTE: ΩʔΛؚΊͯ4KB·Ͱ
    environment {
    variables = {
    BQ_CREDENTIALS = "${data.aws_kms_ciphertext.bq_credentials.ciphertext_blob}"
    BQ_PROJECT = "${var.google["project"]}"
    BQ_DATASET = "${google_bigquery_dataset.main.dataset_id}"
    BQ_TABLE = "${google_bigquery_table.main.table_id}"
    }
    }
    }
    resource "aws_lambda_event_source_mapping" "kinesis2bq" {
    batch_size = 100 # default 100
    event_source_arn = "${aws_kinesis_stream.main.arn}"
    function_name = "${aws_lambda_function.kinesis2bq.arn}"
    starting_position = "TRIM_HORIZON"
    }
    

    View Slide

30. data "aws_kms_ciphertext" "bq_credentials" {
    key_id = "${aws_kms_key.main.id}"
    plaintext = "${file(var.bq["credentials"])}"
    }
    variable "bq" {
    default = {
    table_id = "myapp"
    credentials = "credentials.json" # }
    }
    

    View Slide

31. resource "aws_iam_role" "kinesis_lambda" {
    name = "${var.service_name}-${terraform.workspace}-kinesis_lambda"
    assume_role_policy = "${data.aws_iam_policy_document.kinesis_lambda_assume_role_policy.json}"
    }
    data "aws_iam_policy_document" "kinesis_lambda_assume_role_policy" {
    statement {
    actions = [
    "sts:AssumeRole",
    ]
    principals {
    type = "Service"
    identifiers = ["lambda.amazonaws.com"]
    }
    }
    }
    resource "aws_iam_role_policy" "kinesis_lambda" {
    name = "${var.service_name}-${terraform.workspace}-kinesis_lambda"
    role = "${aws_iam_role.kinesis_lambda.id}"
    policy = "${data.aws_iam_policy_document.kinesis_lambda_role_policy.json}"
    }
    data "aws_iam_policy_document" "kinesis_lambda_role_policy" {
    statement {
    actions = ["kinesis:*"]
    resources = ["${aws_kinesis_stream.main.arn}"]
    }
    statement {
    actions = ["kms:Decrypt"]
    resources = ["${aws_kms_key.main.arn}"]
    }
    statement {
    actions = [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:PutLogEvents",
    ]
    resources = [
    "arn:aws:logs:*:*:*",
    ]
    }
    }
    

    View Slide

32. 1 import base64
    2 import datetime
    3 import json
    4 import os
    5 import sys
    6
    7 import boto3
    8
    9 sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'vendor'))
    10 from gcloud import bigquery
    11 from oauth2client.service_account import ServiceAccountCredentials
    12
    13 BQ_CREDENTIALS = os.environ['BQ_CREDENTIALS']
    14 BQ_PROJECT = os.environ['BQ_PROJECT']
    15 BQ_DATASET = os.environ['BQ_DATASET']
    16 BQ_TABLE = os.environ['BQ_TABLE']
    17
    18 def handler(event, context):
    19 rows = []
    20
    21 for r in event['Records']:
    22 payload = r['kinesis']['data']
    23 try:
    24 data = json.loads(base64.b64decode(payload))
    25 row = []
    26 for key in ['time', 'tag', 'value']:
    27 if key == 'time':
    28 row.append(datetime.datetime.fromtimestamp(data[key]))
    29 else:
    30 row.append(data[key])
    31 rows.append(tuple(row))
    32 except Exception as e:
    33 print('Invalid data "{0}": {1}'.format(payload, e))
    34 pass
    35
    36 if len(rows) == 0:
    37 return
    38
    39 kms = boto3.client('kms')
    40 blob = base64.b64decode(BQ_CREDENTIALS)
    41 dec = kms.decrypt(CiphertextBlob = blob)
    42 keyfile_dict = json.loads(dec['Plaintext'])
    43 credentials = ServiceAccountCredentials.from_json_keyfile_dict(keyfile_dict)
    44
    45 bq = bigquery.Client(credentials = credentials, project = BQ_PROJECT)
    46 dataset = bq.dataset(BQ_DATASET)
    47 table = dataset.table(BQ_TABLE)
    48 table.reload()
    49 res = table.insert_data(rows)
    50
    51 print(res)
    

    View Slide

33. ߏங
    $ pip install gcloud -t lambda/kinesis2bq/vendor/
    $ test -e lambda/kinesis2bq/vendor/google/__init__.py || \
    > touch lambda/kinesis2bq/vendor/google/__init__.py
    $ terraform init
    $ terraform plan --var-file=config.tfvars
    $ terraform apply --var-file=config.tfvars
    $ terraform show
    ࣄલ४උʢ3&"%.&NEࢀরʣ
    ߏஙͱ֬ೝ
    

    View Slide

34. ಈ࡞֬ೝ
    $ TEST_JSON=$(mktemp)
    $ test/gen_kinesis_records.sh $KINESIS_STREAM >$TEST_JSON
    $ cat $TEST_JSON | jq . -C | less -R
    $ aws kinesis put-records --cli-input-json file://$TEST_JSON
    ,JOFTJTʹσʔλΛ౤ೖ
    #JH2VFSZͰσʔλΛ֬ೝ
    $ bq query "SELECT COUNT(time) FROM $BIGQUERY"
    ˞,*/&4*4@453&".ͱ#*(26&3:͸UFSSBGPSNBQQMZͷ݁ՌͰஔ͖׵͑
    

    View Slide

35. ·ͱΊ
    

    View Slide

36. ·ͱΊ
    w ଟ͘ͷϓϥοτϑΥʔϜ΍αʔϏεΛαϙʔτͨ͠ɺΠϯϑ
    ϥͷϓϩϏδϣχϯάπʔϧ
    w ΠϯϑϥશମΛίʔυͰهड़͠ɺόʔδϣϯ؅ཧɺڞ༗ɺ࠶
    ར༻Ͱ͖Δ
    w ϓϥοτϑΥʔϜ͝ͱͷಠཱͨ͠πʔϧΛ࢖Θͳͯ͘ྑ͍
    w ࣄલͷ࣮ߦܭըͷ֬ೝʹΑΓෆҙͷมߋΛ๷͙͜ͱ͕Ͱ͖Δ
    

    View Slide