Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GCP+GKE Deep Dive Part 1: Initial App Development

GCP+GKE Deep Dive Part 1: Initial App Development

Part 2 바로가기: https://speakerdeck.com/premist/gcp-plus-gke-deep-dive-part-2-advanced-cluster-management/

처음 GKE를 사용하기 시작하면 난해할 수 있는 서비스 배포, GKE 클러스터를 생성하고 여러 GCP의 서비스를 이용하여 첫 애플리케이션을 배포하는 과정까지 자세하게 살펴봅니다. 또한 애플리케이션을 배포한 이후 안정적인 서비스 운영을 위해 활용할 수 있는 클러스터 관리 테크닉을 소개합니다.

Part 1: Initial App Deployment
Google Kubernetes Engine에 처음 애플리케이션을 배포할 때, 어디부터 시작해야 하는지, 클러스터를 생성하고 설정할 때 주의할 점은 무엇인지 난해한 경우가 많습니다. 또한 Google Cloud에서 제공하는 여러 서비스를 적절히 활용하려고 해도 사용 사례나 튜토리얼을 찾아보는 것에도 한계가 있기 마련입니다.

본 세션의 첫 번째 파트에서는 소스 코드 호스팅 및 협업 애플리케이션인 GitLab CE Omnibus를 Google Kubernetes Engine(GKE)에 배포하는 예시를 통해 적당히 규모가 있는 애플리케이션을 GKE에 배포하는 전략을 알아봅니다. 또한 PostgreSQL과 Redis를 GKE에 직접 호스팅하는 대신, Google Cloud에서 매니지드 형태로 제공하는 서비스인 Cloud SQL과 Cloud Memorystore 인스턴스를 각각 생성하고, GKE 내에서 각각의 서비스에 연결하는 방법을 자세하게 설명합니다.

필요 이해도: 컨테이너에 대한 이해와 통상적인 애플리케이션 배포 과정에 대한 이해를 전제로 진행되고, Google Cloud SDK와 Kubernetes CLI를 사용하므로 커맨드 라인 도구의 사용에 능숙한 경우 세션의 내용을 보다 쉽게 이해하실 수 있습니다.

Minku Lee

June 29, 2018
Tweet

More Decks by Minku Lee

Other Decks in Programming

Transcript

  1. PART 1 13:00~
    Initial App Deployment

    View Slide

  2. 팖뼣켆푢 !

    View Slide

  3. PART 1 Initial App Deployment
    GitLab CEܳ GCP+GKEী ߓನ೧ࠇद׮
    Advanced Cluster Management
    ৈ۞ ௿۞झఠ ਍৔ ప௼ץਸ ࣗѐ೤פ׮
    PART 2
    13:00-13:50
    14:00-14:50

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. Google Kubernetes Engine펞
    얺큲캫컿쭎컪찒큲짾밚힎

    View Slide

  8. kubernetes

    View Slide

  9. 핂뻖큲흂잏
    Container Scheduling
    핞솧핺쫃묺
    Auto-healing
    컪찒큲싢큲쩒읺
    Service Discovery
    컲헣뫎읺
    Config Management
    쭎쭒칾

    Load Balancing
    슿슿˘
    kubernetes

    View Slide

  10. View Slide

  11. Kubernetes Engine

    View Slide

  12. View Slide

  13. 픒짾쫓킪삲

    View Slide

  14. View Slide

  15. View Slide

  16. Open Source
    Ruby on Rails엖핒풚
    PostgreSQL섾핂쩮핂큲
    Redis핟펓짝킪섾핂쩮핂큲
    Community Edition

    View Slide

  17. Cloud SQL
    FOR POSTGRESQL
    Cloud Memorystore
    FOR REDIS

    View Slide

  18. ✅ 훎옪힎풞
    ✅ 잲삖힎슪컪찒큲
    ✅ 큲핊잏힎풞
    ✅ High Availability묺컿힎풞
    Cloud SQL Cloud Memorystore

    View Slide

  19. GitLab
    Deployment
    Multiple Pods
    GitLab
    Deployment
    Multiple Pods
    GitLab
    Service
    GitLab
    Ingress
    GLBC
    Cloud HTTP(S) Load Balancer
    Postgres
    Cloud SQL Instance
    Redis
    Cloud Memorystore Instance
    Kubernetes Engine
    Google Cloud Platform

    View Slide

  20. 킪핟믾헒
    • 핂뻖펞샎핂퐎캏헏핆팮읺핂켦짾뫊헣펞샎
    핂읊헒헪옪삖삲
    ˖ 잶슪않핆솒묺픦칺푷픒믾쫆픊옪삖삲
    • Google Cloud SDK (gcloud)짝

    Kubernetes CLI (kubectl) 픒칺푷삖삲

    View Slide

  21. GitLab
    Deployment
    Multiple Pods
    GitLab
    Deployment
    Multiple Pods
    GitLab
    Service
    GitLab
    Ingress
    GLBC
    Cloud HTTP(S) Load Balancer
    Postgres
    Cloud SQL Instance
    Redis
    Cloud Memorystore Instance
    Kubernetes Engine
    Google Cloud Platform

    View Slide

  22. View Slide

  23. $ gcloud container clusters create hello-gke \
    --project=shakr-openinfra-demo \
    --zone=asia-northeast1-b \
    --cluster-version=1.10.4-gke.2 \
    --machine-type=n1-standard-1 \
    --num-nodes=3 \
    --enable-ip-alias \
    --enable-autorepair

    View Slide

  24. --enable-autorepair
    쫃묺많푢Node많핖픒쌚핞솧픊옪쫃묺쿦
    ˖ Node Health Check많킲쁢몋푾
    ˖ Node많status쫂몮읊힎팘쁢몋푾
    ˖ Node픦쭎싢큲핢펺푷얗핂펔쁢몋푾

    View Slide

  25. --enable-ip-alias
    VPC뻲풚픦CIDR쯢옫픒칺푷펺Pod IP읊샇
    ˖ GKE퐎VPC많컪옪맧픎CIDR쯢옫픒칺푷쿦핖픚
    ˖ GKE Cluster 짤펞컪솒VPC 뺂않졂 Pod IP펞헟믊많쁳
    ˖ Proxy 펔핂Cloud Memorystore헟믊많쁳

    View Slide

  26. $ gcloud container clusters create hello-gke \
    --project=shakr-openinfra-demo \
    --zone=asia-east1-b \
    --cluster-version=1.10.4-gke.2 \
    --machine-type=n1-standard-1 \
    --num-nodes=3 \
    --enable-ip-alias \
    --enable-autorepair

    View Slide

  27. --project=shakr-openinfra-demo \
    --zone=asia-east1-b \
    --cluster-version=1.10.4-gke.2 \
    --machine-type=n1-standard-1 \
    --num-nodes=3 \
    --enable-ip-alias \
    --enable-autorepair
    Creating cluster hello-gke...done.
    Created [https://container.googleapis.com/v1/projects/shakr-
    openinfra-demo/zones/asia-northeast1-b/clusters/hello-gke].
    NAME LOCATION MASTER_VERSION MASTER_IP
    hello-gke asia-northeast1-b 1.10.4-gke.2 35.200.25.152

    View Slide

  28. GitLab
    Deployment
    Multiple Pods
    GitLab
    Deployment
    Multiple Pods
    GitLab
    Service
    GitLab
    Ingress
    GLBC
    Cloud HTTP(S) Load Balancer
    Postgres
    Cloud SQL Instance
    Redis
    Cloud Memorystore Instance
    Kubernetes Engine
    Google Cloud Platform

    View Slide

  29. $ gcloud sql instances create gitlab-postgresql \
    --availability-type=regional \
    --cpu=1 --memory=4GiB \
    --database-version=POSTGRES_9_6 \
    --region=asia-east1 \
    --storage-size=10GB \
    --storage-type=SSD \
    --storage-auto-increase
    $ gcloud sql users create gitlab % \
    --instance=gitlab-postgresql --password=mySecurePassword!
    Cloud SQL핆큲큲캫컿

    View Slide

  30. $ gcloud alpha redis instances create gitlab-redis \
    --region=asia-east1 \
    --size=2
    Cloud Memorystore핆큲큲캫컿

    View Slide

  31. Service Account

    View Slide

  32. Service Account
    • GCP컪찒큲펞헟믊쁢맏팮읺핂켦픦킮풞픒샎
    • 읺콚큲쪒뭚쭎펺많쁳
    • JSON킫픦읊슫펺칺푷
    • GKE Pod펞컪Cloud SQL헟믊킪칺푷

    View Slide

  33. $ gcloud iam service-accounts create gitlab \
    --display-name="GitLab Service Account"
    Service Account캫컿

    View Slide

  34. $ gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:$EMAIL"\
    --role="roles/cloudsql.client"
    # Service Accountী Cloud Storage Admin Roleਸ ೡ׼
    $ gcloud projects add-iam-policy-binding $PROJECT \
    --member="serviceAccount:$EMAIL"\
    --role="roles/storage.admin"
    뭚쭎펺

    View Slide

  35. $ gcloud iam service-accounts keys create \
    ./artifacts/serviceaccount.json \
    --iam-account $EMAIL

    View Slide

  36. {
    "type": "service_account",
    "project_id": "shakr-openinfra-demo",
    "private_key_id": "1234567890abcdef1234567890",
    "private_key": "-----BEGIN PRIVATE KEY-----\n....\n-----END
    PRIVATE KEY-----\n",
    "client_email": "[email protected]
    demo.iam.gserviceaccount.com",
    "client_id": "12345678901234567890",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://accounts.google.com/o/oauth2/token",
    JSON

    View Slide

  37. Pod & Deployment

    View Slide

  38. GitLab
    Deployment
    Multiple Pods
    GitLab
    Deployment
    Multiple Pods
    GitLab
    Service
    GitLab
    Ingress
    GLBC
    Cloud HTTP(S) Load Balancer
    Postgres
    Cloud SQL Instance
    Redis
    Cloud Memorystore Instance
    Kubernetes Engine
    Google Cloud Platform

    View Slide

  39. Pod
    • Kubernetes뺂많핳핟픎픦삶퓒
    • 빦픎펺얺맪픦핂뻖옪묺컿
    • Pod팖픦핂뻖쁢헎핳뫃맒슿픒뫃퓮
    • 믾쫆헏픊옪홓욚킪졶슮섾핂많칻헪쇶
    • 푢킪PersistentVolume슿픒칺푷펺핊퓮힎많쁳

    View Slide

  40. Deployment
    • Instance Group(Auto-Scaling Group)뫊맧픎맪뼞
    • Pod픒짾몮뫎읺훚
    • replica읊힎헣훊졂믆쿦잚Pod픒많픎칻헪
    • Rolling update 짝 rollback 힎풞

    View Slide

  41. apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: gitlab
    labels:
    app: gitlab
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: gitlab
    template:
    metadata:
    labels:
    apps: gitlab
    spec:
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    resources:
    requests:
    cpu: "0.5"
    memory: 1Gi
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    value: ...
    deployment.yml
    Deployment
    spec
    Pod spec

    View Slide

  42. spec:
    containers:
    - name: gitlab
    deployment.yml (pod spec)
    - name: cloudsql-proxy

    View Slide

  43. View Slide

  44. Cloud SQL Proxy
    • Cloud SQL픎Memorystore퐎삲읂멚VPC IP옪짢옪헟믊쿦
    펔몮몮헣*1샎펻픒whitelisting쁢짷킫핂않GKE얺큲펞컪
    칺푷핂삲콚쭖
    • Cloud SQL Proxy읊핂푷졂Service Account픦Cloud SQL
    Client Role옪옪옫킪읊폂쿦핖삲
    • Google펞컪맪짪펺짢핂뻖읺짝Docker핂짆힎짾

    View Slide

  45. View Slide

  46. spec:
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    deployment.yml (pod spec)
    - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy
    command:
    ["/cloud_sql_proxy", "-instances=..."]

    View Slide

  47. spec:
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    value: ...
    deployment.yml (pod spec)
    - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy
    command:
    ["/cloud_sql_proxy", "-instances=..."]
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: ...
    #

    View Slide

  48. $

    View Slide

  49. Secret

    View Slide

  50. Secret
    • Kubernetes얺큲펞컪짊맞헣쫂읊헎핳쌚칺푷
    • GKE샎킪쫂슪펞컪쿶멶힞읺
    • 짊맞힎팘픎몋쪎쿦슿픒헎핳쌚쁢찒킅묺혾핆ConfigMap
    픒칺푷

    View Slide

  51. View Slide

  52. apiVersion: v1
    kind: Secret
    metadata:
    name: my-secrets
    type: Opaque
    data:
    GOOGLE_CLOUD_KEYFILE_JSON:
    SENTRY_DSN:
    secret.example.yml

    View Slide

  53. apiVersion: v1
    kind: Secret
    metadata:
    name: my-secrets
    type: Opaque
    data:
    GOOGLE_CLOUD_KEYFILE_JSON: eyJseXJpY3MiOiAiV2UncmUgbm8gc3RyYW5nZXJzIHRvIGxvdmUNCllvdSBr

    bm93IHRoZSBydWxlcyBhbmQgc28gZG8gSQ0KQSBmdWxsIGNvbW1pdG1lbnQncyB3aGF0IEknbSB0aGlua2luZyBvZ
    g0KWW91IHdvdWxkbid0IGdldCB0aGlzIGZyb20gYW55IG90aGVyIGd1eQ0KSSBqdXN0IHdhbm5hIHRlbGwgeW91IG
    hvdyBJJ20gZmVlbGluZw0KR290dGEgbWFrZSB5b3UgdW5kZXJzdGFuZA0KTmV2ZXIgZ29ubmEgZ2l2ZSB5b3UgdXA
    NCk5ldmVyIGdvbm5hIGxldCB5b3UgZG93bg0KTmV2ZXIgZ29ubmEgcnVuIGFyb3VuZCBhbmQgZGVzZXJ0IHlvdQ0K
    TmV2ZXIgZ29ubmEgbWFrZSB5b3UgY3J5DQpOZXZlciBnb25uYSBzYXkgZ29vZGJ5ZQ0KTmV2ZXIgZ29ubmEgdGVsb
    CBhIGxpZSBhbmQgaHVydCB5b3UifQ==
    SENTRY_DSN: aHR0cHM6Ly9yaWNrOmFzaGxleUBuZXZlcmdvbm5hZ2l2ZXlvdS51cDo1MzIxNA==
    secret.example.yml

    View Slide

  54. $ kubectl create secret generic gitlab-config \
    --from-literal=redis_host=10.0.0.3 \
    --from-file=./artifacts/gitlab.rb \
    --from-file=./artifacts/serviceaccount.json
    kubectl픒핂푷Secret캫컿

    View Slide

  55. spec:
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    valueFrom:
    secretKeyRef:
    name: gitlab-config
    key: gitlab.rb
    deployment.yml (pod spec)
    - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy
    command:
    ["/cloud_sql_proxy", "-instances=..."]
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /mnt/config/serviceaccount.json
    volumeMounts:
    - name: config
    mountPath: /mnt/config
    readOnly: true
    volumes:
    - name: config
    secret:
    secretName: gitlab-config
    #

    View Slide

  56. Pod
    • Kubernetes뺂많핳핟픎픦삶퓒
    • 빦픎펺얺맪픦핂뻖옪묺컿
    • Pod팖픦핂뻖쁢헎핳뫃맒슿픒뫃퓮
    • 믾쫆헏픊옪홓욚킪졶슮섾핂많칻헪쇶
    • 푢킪PersistentVolume슿픒칺푷펺핊퓮힎많쁳

    View Slide

  57. "GitLab픦 Git 섾핂쁢펂싢펞헎핳쇦힎?"

    View Slide

  58. RTFM

    View Slide

  59. View Slide

  60. View Slide

  61. %

    View Slide

  62. Volume 잖풂픦홓윦
    • Secret
    • ConfigMap
    • gcePersistentDisk
    • hostPath
    • emptyDir
    • persistentVolumeClaim
    • …

    View Slide

  63. $ gcloud compute disks create my-disk \
    --size=10GB \
    --type=pd-ssd \
    --zone asia-east1-a \
    ...
    Compute Engine Persistent Disk 캫컿
    &

    View Slide

  64. Volume 잖풂픦홓윦
    • Secret
    • ConfigMap
    • gcePersistentDisk
    • hostPath
    • emptyDir
    • persistentVolumeClaim
    • …

    View Slide

  65. PersistentVolumeClaim

    View Slide

  66. PersistentVolumeClaim
    • 섾핂많쫂홂쇦쁢싢큲읊 Kubernetes펞푢
    • GCE Persistent Disk읊핞솧픊옪캫컿몮믆펞캏픟쁢
    PersistentVolume픒캫컿
    • StorageClass옪Persistent Disk픦홓윦읊힎헣

    View Slide

  67. View Slide

  68. View Slide

  69. View Slide

  70. apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: ssd
    provisioner: kubernetes.io/gce-pd
    parameters:
    type: pd-ssd
    storageclass.yml

    View Slide

  71. apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
    name: gitlab-data
    spec:
    accessModes:
    - ReadWriteOnce
    storageClassName: ssd
    resources:
    requests:
    storage: 50Gi
    pvc.yml

    View Slide

  72. spec:
    containers:
    - name: gitlab
    image: gitlab/gitlab-ce:latest
    env:
    - name: GITLAB_OMNIBUS_CONFIG
    valueFrom:
    secretKeyRef:
    name: gitlab-config
    key: gitlab.rb
    volumeMounts:
    - name: gitlab-data
    mountPath: /var/opt/gitlab
    volumes:
    - name: gitlab-data
    persistentVolumeClaim:
    claimName: gitlab-data
    deployment-with-pvc.yml (pod spec)
    - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy
    command:
    ["/cloud_sql_proxy", "-instances=..."]
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: /mnt/config/serviceaccount.json
    volumeMounts:
    - name: config
    mountPath: /mnt/config
    readOnly: true
    volumes:
    - name: config
    secret:
    secretName: gitlab-config

    View Slide

  73. 짾

    View Slide

  74. $ kubectl apply -f deployment.yml
    kubectl옪Deployment캫컿

    View Slide

  75. Demo

    View Slide

  76. $ kubectl port-forward POD_NAME 8080:80
    옪Port forwarding proxy 캫컿
    #

    View Slide

  77. "칺푷핞많GitLab펞펂쎉멚헟콛힎

    View Slide

  78. Service & Ingress

    View Slide

  79. Service
    • ౠ੿ Pod ٜਸ Kubernetes 얺큲뺂펞컪펂쎉멚헟믊퍊쁢힎
    헣픦쁢짷쩣
    • NodePort핓Worker Node픦읊샇
    • Loadbalancer핓TCP Load Balancer 캫컿(GCP)
    • Internal/External 졶숞힎풞

    View Slide

  80. $ kubectl expose deployment gitlab \
    --port=80 --target-port=80 --type=NodePort
    kubectl옪Service캫컿

    View Slide

  81. apiVersion: v1
    kind: Service
    metadata:
    name: gitlab
    spec:
    selector:
    app: gitlab
    ports:
    - port: 80
    protocol: TCP
    name: http
    type: NodePort
    service.yml

    View Slide

  82. apiVersion: v1
    kind: Service
    metadata:
    name: gitlab
    spec:
    selector:
    app: gitlab
    ports:
    - port: 80
    protocol: TCP
    name: http
    type: NodePort
    service.yml

    View Slide

  83. apiVersion: v1
    kind: Service
    metadata:
    name: gitlab
    spec:
    selector:
    app: gitlab
    ports:
    - port: 80
    protocol: TCP
    name: http
    - port: 443
    protocol: TCP
    name: https
    type: NodePort
    service-multiport.yml

    View Slide

  84. Ingress
    • Service৬ Public internet픒펾멾훊쁢읺콚큲
    • Service펞type=LoadBalancer읊칺푷졂짦슪킪푢힎쁢팘픚
    • 옲얺많않푾픒샂샇(NGINX, Traefik, …)
    • GKE펞컪쁢Cloud HTTP(S) Load Balancer읊칺푷
    • IPv6/SSL Termination, CDN, HTTP->HTTPS Redirect슿

    삲퍟믾쁳힎풞
    BETA

    View Slide

  85. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    spec:
    backend:
    serviceName: gitlab
    servicePort: 80
    ingress.yml

    View Slide

  86. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    spec:
    tls:
    - secretName: tls-gitlab
    backend:
    serviceName: gitlab
    servicePort: 80
    ingress.yml

    View Slide

  87. $ kubectl create secret tls tls-gitlab \
    --cert=./artifacts/tls/cert.crt \
    --key=./artifacts/tls/key.key
    kubectl옪TLS Secret캫컿

    View Slide

  88. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    spec:
    tls:
    - secretName: tls-gitlab
    backend:
    serviceName: gitlab
    servicePort: 80
    ingress.yml

    View Slide

  89. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    annotations:
    kubernetes.io/ingress.allow-http: "false"
    spec:
    tls:
    - secretName: tls-gitlab
    backend:
    serviceName: gitlab
    servicePort: 80
    ingress.yml
    '

    View Slide

  90. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: "gitlab"
    spec:
    tls:
    - secretName: tls-gitlab
    backend:
    serviceName: gitlab
    servicePort: 80
    ingress.yml

    View Slide

  91. View Slide

  92. $ gcloud compute addresses create gitlab --global
    gcloud옪Regional IP캫컿

    View Slide

  93. Demo

    View Slide

  94. View Slide

  95. apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: gitlab
    namespace: default
    spec:
    tls:
    - secretName: tls-a
    - secretName: tls-b
    rules:
    - host: a.exmaple.com
    http:
    paths:
    - backend:
    serviceName: a
    servicePort: 80
    - host: b.example.com
    http:
    paths:
    - backend:
    serviceName: b
    servicePort: 80
    ingress-advanced.yml

    View Slide

  96. Recap

    View Slide

  97. https://twitter.com/tenderlove/status/988887936128040960

    View Slide

  98. GitLab
    Deployment
    Multiple Pods
    GitLab
    Deployment
    Multiple Pods
    GitLab
    Service
    GitLab
    Ingress
    GLBC
    Cloud HTTP(S) Load Balancer
    Postgres
    Cloud SQL Instance
    Redis
    Cloud Memorystore Instance
    Kubernetes Engine
    Google Cloud Platform

    View Slide

  99. View Slide

  100. View Slide

  101. View Slide

  102. View Slide

  103. View Slide

  104. View Slide

  105. View Slide

  106. 맞칺삖삲 (

    View Slide

  107. GCP+GKE Deep Dive
    Minku Lee
    CTO Shakr
    Shakr펞컪쁳엳핖쁢펢힎삖펂읊졶킻삖삲
    careers.shakr.com

    View Slide