안드로이드 개발 환경 개선을 위한 노력들

#Android #CI/CD #Build

개발 환경 개선을 위해 안드팀과 데브옵스팀이 함께 도전했던 Jenkins CI, RemoteBuild를 소개합니다.

발표자 : smith.jspiner
안드로이드 외에도 다양한 분야에 관심이 많은 개발자 smith입니다. 서비스 개발과 더불어 프로세스 개선에도 큰 관심을 두고 있습니다.

louis.0802
카카오모빌리티에서 devops 엔지니어로 일하고 있는 louis입니다. 개발과 인프라 관리 프로세스의 자동화 및 간소화에 관심이 많습니다.

june.bee
더 효율적인 환경, 더 안정적인 서비스를 위하여 다방면으로 고민이 많은 Devops 엔지니어 june입니다.

Transcript

1. Jenkins CI 파이프라인과 리모트 빌드 소개
   Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   안드로이드 개발 환경 개선을 위한 노력들
   정성민 Smith.JSpiner, 이재욱 Louis.0802, 소준상 June.Bee
   
   
   카카오모빌리티
   if(kakao)2022
   

   View Slide

2. Jenkins CI 파이프라인과 리모트 빌드 소개
   Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   안드로이드 개발 환경 개선을 위한 노력들
   정성민 Smith.JSpiner, 이재욱 Louis.0802, 소준상 June.Bee
   
   
   카카오모빌리티
   if(kakao)2022
   

   View Slide

3. branch 안정성을 검증 해주는 Jenkins CI 구축과정
   
   
   빠른 빌드를 도와주는 리모트 빌드
   
   
   향후 계획
   

   View Slide

4. branch 안정성을 검증 해주는 Jenkins CI 구축과정
   
   
   빠른 빌드를 도와주는 리모트 빌드
   
   
   향후 계획
   

   View Slide

5. develop
   feature/new1
   feature/new2
   

   View Slide

6. develop
   feature/new1
   feature/new2
   

   View Slide

7. Pull
   Request
   Merge
   Review
   

   View Slide

8. Pull
   Request
   Merge
   Review
   

   View Slide

9. compile test lint
   

   View Slide

10. develop
    feature/new
    

    View Slide

11. develop
    feature/new
    1. 문제가 된 코드 찾기
    
    
    2. 누가 작성했는지 or 어느 팀 담당인지 찾기
    
    
    3. 해당팀에 수정요청
    
    
    4. 수정이 완료된후 이후 작업 진행
    

    View Slide

12. develop
    feature/new
    1. 문제가 된 코드 찾기
    
    
    2. 누가 작성했는지 or 어느 팀 담당인지 찾기
    
    
    3. 해당팀에 수정요청
    
    
    4. 수정이 완료된후 이후 작업 진행
    

    View Slide

13. Build #111
    Build #112
    Build #113
    Jenkins
    

    View Slide

14. Build #111
    Build #112
    Build #113
    Jenkins
    

    View Slide

15. pull request
    PR Pipeline
    - PR 템플릿 체크
    
    
    - debug compile
    
    
    - debug test
    
    
    - sync GitHub status
    sync state
    error alert
    1. PR PIPELINE
    

    View Slide

16. pull request
    PR Pipeline
    - PR 템플릿 체크
    
    
    - debug compile
    
    
    - debug test
    
    
    - sync GitHub status
    sync state
    error alert
    1. PR PIPELINE
    

    View Slide

17. branch update
    PUSH Pipeline
    - release Compile
    
    
    - lint
    
    
    - PR pipeline
    error alert
    2. PUSH PIPELINE
    

    View Slide

18. branch update
    PUSH Pipeline
    - release Compile
    
    
    - lint
    
    
    - PR pipeline
    error alert
    2. PUSH PIPELINE
    

    View Slide

19. 빌드 캐시
    Container preset
    Remote Cache Server
    Jenkins
    Jenkins
    

    View Slide

20. 빌드 캐시
    Container preset
    Remote Cache Server
    Jenkins
    Jenkins
    

    View Slide

21. View Slide

22. branch 안정성을 검증 해주는 Jenkins CI 구축과정
    
    
    빠른 빌드를 도와주는 리모트 빌드
    
    
    향후 계획
    

    View Slide

23. 환경 빌드시간
    증분 빌드
    최소 3 s
    평균 60s
    최대 3m 47s
    풀 빌드
    최소 14m 30s
    평균 15m 23s
    최대 16m 7s
    빈도
    소요시간(초)
    3 99 195 388 285 581 870 967
    로컬 빌드
    

    View Slide

24. 환경 빌드시간
    증분 빌드
    최소 3 s
    평균 60s
    최대 3m 47s
    풀 빌드
    최소 14m 30s
    평균 15m 23s
    최대 16m 7s
    빈도
    소요시간(초)
    3 99 195 388 285 581 870 967
    로컬 빌드
    

    View Slide

25. multi module gradle setting
    remote
    
    
    build cache
    

    View Slide

26. multi module gradle setting
    remote
    
    
    build cache
    

    View Slide

27. local machine(맥북)
    remote build server
    1. upload
    2. execute
    소스코드
    apk, test result..
    후킹
    3. download
    Mirakle
    rsync
    

    View Slide

28. local machine(맥북)
    remote build server
    1. upload
    2. execute
    소스코드
    apk, test result..
    후킹
    3. download
    Mirakle
    rsync
    

    View Slide

29. local machine(맥북)
    remote build server
    1. upload
    2. execute
    소스코드
    apk, test result..
    후킹
    3. download
    Mirakle
    rsync
    

    View Slide

30. local machine mac mini
    internal network
    nexus proxy
    Mirakle
    

    View Slide

31. local machine mac mini
    internal network
    nexus proxy
    Mirakle
    

    View Slide

32. local machine mac mini
    internal network
    nexus proxy
    Mirakle
    

    View Slide

33. DEVELOPER
    OFFICE
    HIGH
    
    
    PERFORMANCE
    요구사항 정리
    PRIVATE
    
    
    NETWORK
    BUILD SERVER
    SIMPLE
    

    View Slide

34. DEVELOPER
    OFFICE
    HIGH
    
    
    PERFORMANCE
    요구사항 정리
    PRIVATE
    
    
    NETWORK
    BUILD SERVER
    SIMPLE
    

    View Slide

35. DEVELOPER
    OFFICE
    HIGH
    
    
    PERFORMANCE
    요구사항 정리
    PRIVATE
    
    
    NETWORK
    BUILD SERVER
    SIMPLE
    

    View Slide

36. DEVELOPER
    OFFICE INFRA TEAM
    어디에, 어떻게 구성?
    PRIVATE
    
    
    NETWORK
    DATA CENTER
    DEVOPS
    CLOUD
    

    View Slide

37. DEVELOPER
    OFFICE INFRA TEAM
    어디에, 어떻게 구성?
    PRIVATE
    
    
    NETWORK
    DATA CENTER
    DEVOPS
    CLOUD
    

    View Slide

38. SSH TUNNEL
    VM
    처음엔 고사양 VM으로 구성
    

    View Slide

39. SSH TUNNEL
    VM
    처음엔 고사양 VM으로 구성
    

    View Slide

40. VM 1
    반복되는 작업, 유저 리소스를 관리해야하는 번거로움
    VM 2 VM 3
    

    View Slide

41. VM 1
    반복되는 작업, 유저 리소스를 관리해야하는 번거로움
    VM 2 VM 3
    

    View Slide

42. VM 1
    반복되는 작업, 유저 리소스를 관리해야하는 번거로움
    VM 2 VM 3
    

    View Slide

43. KUBERNETES
    쿠버네티스를 활용
    POD
    POD
    POD
    

    View Slide

44. KUBERNETES
    쿠버네티스를 활용
    POD
    POD
    POD
    

    View Slide

45. KUBERNETES
    쿠버네티스를 활용
    POD
    POD
    POD
    

    View Slide

46. 리모트 빌드 인프라 환경 구성
    POD
    PRIVATE KEY PUBLIC KEY
    PORT FORWARD
    

    View Slide

47. 리모트 빌드 인프라 환경 구성
    POD
    PRIVATE KEY PUBLIC KEY
    PORT FORWARD
    

    View Slide

48. 리모트 빌드 인프라 환경 구성
    POD
    PRIVATE KEY PUBLIC KEY
    PORT FORWARD
    

    View Slide

49. 하지만, 반복되는 작업은 아직도 존재
    JOB
    JOB
    JOB
    DEVOPS
    TOO MANY INFO
    TOO MANY JOB
    JOB
    

    View Slide

50. 하지만, 반복되는 작업은 아직도 존재
    JOB
    JOB
    JOB
    DEVOPS
    TOO MANY INFO
    TOO MANY JOB
    JOB
    

    View Slide

51. 하지만, 반복되는 작업은 아직도 존재
    JOB
    JOB
    JOB
    DEVOPS
    TOO MANY INFO
    TOO MANY JOB
    JOB
    

    View Slide

52. 많은 수요
    자동화
    많은 커맨드
    많은 작업
    

    View Slide

53. 많은 수요
    자동화
    많은 커맨드
    많은 작업
    

    View Slide

54. 많은 수요
    자동화
    많은 커맨드
    많은 작업
    

    View Slide

55. 많은 수요
    자동화
    많은 커맨드
    많은 작업
    

    View Slide

56. 많은 수요
    자동화
    많은 커맨드
    많은 작업
    

    View Slide

57. git clone $GIT_REPO $TEMP_DIR
    cp $REMOTEBUILD_DIR/sample/sample-deploy.yaml $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    replace "s/username$/${BUILDER}/g" $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    git push origin master
    argocd
    

    View Slide

58. git clone $GIT_REPO $TEMP_DIR
    cp $REMOTEBUILD_DIR/sample/sample-deploy.yaml $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    replace "s/username$/${BUILDER}/g" $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    git push origin master
    argocd
    

    View Slide

59. git clone $GIT_REPO $TEMP_DIR
    cp $REMOTEBUILD_DIR/sample/sample-deploy.yaml $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    replace "s/username$/${BUILDER}/g" $REMOTEBUILD_DIR/$APP_NAME/$BUILDER-deploy.yaml
    git push origin master
    argocd
    

    View Slide

60. <..CREATE MAINFRAIMER FILE..>
    echo "$SSH_DIR/remotebuild/id_rsa" | ssh-keygen -m PEM -t rsa -b 4096 -P ''
    POD_NAME=
    POD_SSH_DIR=“/home/devopsuser/.ssh/authorized_keys"
    kubectl cp $SSH_DIR/remotebuild/id_rsa.pub $POD_NAME:$POD_SSH_DIR --namespace $NAMESPACE
    kubectl exec -it $POD_NAME -- bash -c "chown devopsuser: $POD_SSH_DIR"
    SSH
    Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    

    View Slide

61. <..CREATE MAINFRAIMER FILE..>
    echo "$SSH_DIR/remotebuild/id_rsa" | ssh-keygen -m PEM -t rsa -b 4096 -P ''
    POD_NAME=
    POD_SSH_DIR=“/home/devopsuser/.ssh/authorized_keys"
    kubectl cp $SSH_DIR/remotebuild/id_rsa.pub $POD_NAME:$POD_SSH_DIR --namespace $NAMESPACE
    kubectl exec -it $POD_NAME -- bash -c "chown devopsuser: $POD_SSH_DIR"
    SSH
    Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    

    View Slide

62. <..CREATE MAINFRAIMER FILE..>
    echo "$SSH_DIR/remotebuild/id_rsa" | ssh-keygen -m PEM -t rsa -b 4096 -P ''
    POD_NAME=
    POD_SSH_DIR=“/home/devopsuser/.ssh/authorized_keys"
    kubectl cp $SSH_DIR/remotebuild/id_rsa.pub $POD_NAME:$POD_SSH_DIR --namespace $NAMESPACE
    kubectl exec -it $POD_NAME -- bash -c "chown devopsuser: $POD_SSH_DIR"
    SSH
    Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    

    View Slide

63. Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    Port
    -
    forwarding
    
    
    
    LOCAL_PORT=2222
    TITLE="PORT FORWARDING TO YOUR POD. CHOOSE ALLOCATED POD."
    POD_NAME=
    kubectl port-forward --namespace $NAMESPACE $POD_NAME $LOCAL_PORT:22
    

    View Slide

64. Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    Port
    -
    forwarding
    
    
    
    LOCAL_PORT=2222
    TITLE="PORT FORWARDING TO YOUR POD. CHOOSE ALLOCATED POD."
    POD_NAME=
    kubectl port-forward --namespace $NAMESPACE $POD_NAME $LOCAL_PORT:22
    

    View Slide

65. Local (127.0.0.1)
    

    
    Remote Build Server
    
    
    
    Port
    -
    forwarding
    
    
    
    LOCAL_PORT=2222
    TITLE="PORT FORWARDING TO YOUR POD. CHOOSE ALLOCATED POD."
    POD_NAME=
    kubectl port-forward --namespace $NAMESPACE $POD_NAME $LOCAL_PORT:22
    

    View Slide

66. ================================================================================
    
    
    # # #### ##### # #### ##### ####
    
    
    ## ## # # # # # # # # # #
    
    
    # ## # # # ##### # # # # # ####
    
    
    # # # # # # # # # ##### #
    
    
    # # # # # # # # # # # #
    
    
    # # #### ##### # #### # ####
    
    
    --------------------------------------------------------------------------------
    
    
    Usage:
    
    
    mobiops [command] [options]
    
    
    Available Commands:
    
    
    --------------- GCP ------------------------------------------------------
    
    
    login ࢶఖೡࣻ ੓ח ҅੿ ܻझ౟ܳ ઑഥ೤פ׮.
    
    
    -n, --new ࢜۽਍ ҅੿ਵ۽ ۽Ӓੋ ೤פ׮.
    
    
    -g, --grep Ѩ࢝ೡ gcp ೐۽ં౟੉ܴਸ ੑ۱೤פ׮. grep -E option
    
    
    c, configure GKE ী ੽Ӕ೤פ׮.
    
    
    -r, --refresh GKE ੽Ӕ ݾ۾ਸ јन೤פ׮.
    
    
    p, project ࢜۽਍ ೐۽ં౟ܳ оܻఈפ׮.
    
    
    k, kube അ੤ ۽Ӓੋೠ ҅੿ਵ۽ ੽Ӕೡ ࣻ ੓ח GKE credential ਸ оઉ২פ׮.
    
    
    cp-secret secret ਸ copy ೤פ׮.
    
    
    usage : /usr/local/bin/mobiops cp-secret SECRET_NAME OLD_Namespace NEW_Namespace
    
    
    w, workload, workid workload identity ܳ োѾ೤פ׮. ksa, gsa োѾ
    
    
    s, ssh GCE ੋझఢझী IAPܳ ా೧ SSH ੽ࣘ೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    r, rdp GCE ੋझఢझী RDP ੽ࣘਸ ਤ೧ IAPܳ ాೠ ఠօਸ प೯೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    --------------- K8S ------------------------------------------------------
    
    
    ns, namespace namespace ܳ ࣇ౴೤פ׮.
    
    
    x, context context ܳ ࣇ౴೤פ׮. (kubectx प೯)
    
    
    b, build ࣗझ௏٘ܳ ࠽٘ ೤פ׮. (default: ./gradlew build -x test)
    
    
    -s, --spring spring build (./gradlew build -x test)
    
    
    d, docker Docker build and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    dx, dockerx Docker buildx amd64 image and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    a, apply അ੤ cluster ী ߓನ೤פ׮.
    
    
    (kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. default : k8s/gcp/)
    
    
    delete, remove അ੤ cluster ী ߓನػ ղਊਸ ࢏ઁ ೤פ׮. (kubectl delete -k )
    
    
    -p, --path kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. (default : k8s/gcp/)
    
    
    l, log, logs pod ۽Ӓܳ ୶੸೤פ׮.
    
    
    e, exec kubectl exec -it {pod}
    
    
    cicd, jenkins cicd ܳ ਤೠ ౵ੌ ࣇ౴
    
    
    - Jenkinsfile ਸ ׮਍߉ই അ੤ ٣۩షܻী ݏ୾ ࣇ౴೤פ׮.
    
    
    - Dockerfile ੉ হਵݶ మ೒݁ਸ ׮਍߉णפ׮.
    
    
    - kustomization directory మ೒݁ਸ ׮਍߉णפ׮.
    
    
    ------------ Android ------------------------------------------------------
    
    
    rb-setenv remotebuildী ೙ਃೠ ssh keyܳ ࢤࢿೞҊ, podী public keyܳ ١۾೤פ׮.
    
    
    rb-portforward remotebuild podী ssh োѾਸ ਤೠ ఠօ݂ਸ ݛযસפ׮.
    
    
    rb-createpod नӏ Userо ࢎਊೡ remotebuild ਊ podܳ ࢤࢿ೤פ׮.
    
    
    ================================================================================
    
    
    KAKAO MOBILITY DEVOPS
    

    MOBIOPS
    

    View Slide

67. ================================================================================
    
    
    # # #### ##### # #### ##### ####
    
    
    ## ## # # # # # # # # # #
    
    
    # ## # # # ##### # # # # # ####
    
    
    # # # # # # # # # ##### #
    
    
    # # # # # # # # # # # #
    
    
    # # #### ##### # #### # ####
    
    
    --------------------------------------------------------------------------------
    
    
    Usage:
    
    
    mobiops [command] [options]
    
    
    Available Commands:
    
    
    --------------- GCP ------------------------------------------------------
    
    
    login ࢶఖೡࣻ ੓ח ҅੿ ܻझ౟ܳ ઑഥ೤פ׮.
    
    
    -n, --new ࢜۽਍ ҅੿ਵ۽ ۽Ӓੋ ೤פ׮.
    
    
    -g, --grep Ѩ࢝ೡ gcp ೐۽ં౟੉ܴਸ ੑ۱೤פ׮. grep -E option
    
    
    c, configure GKE ী ੽Ӕ೤פ׮.
    
    
    -r, --refresh GKE ੽Ӕ ݾ۾ਸ јन೤פ׮.
    
    
    p, project ࢜۽਍ ೐۽ં౟ܳ оܻఈפ׮.
    
    
    k, kube അ੤ ۽Ӓੋೠ ҅੿ਵ۽ ੽Ӕೡ ࣻ ੓ח GKE credential ਸ оઉ২פ׮.
    
    
    cp-secret secret ਸ copy ೤פ׮.
    
    
    usage : /usr/local/bin/mobiops cp-secret SECRET_NAME OLD_Namespace NEW_Namespace
    
    
    w, workload, workid workload identity ܳ োѾ೤פ׮. ksa, gsa োѾ
    
    
    s, ssh GCE ੋझఢझী IAPܳ ా೧ SSH ੽ࣘ೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    r, rdp GCE ੋझఢझী RDP ੽ࣘਸ ਤ೧ IAPܳ ాೠ ఠօਸ प೯೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    --------------- K8S ------------------------------------------------------
    
    
    ns, namespace namespace ܳ ࣇ౴೤פ׮.
    
    
    x, context context ܳ ࣇ౴೤פ׮. (kubectx प೯)
    
    
    b, build ࣗझ௏٘ܳ ࠽٘ ೤פ׮. (default: ./gradlew build -x test)
    
    
    -s, --spring spring build (./gradlew build -x test)
    
    
    d, docker Docker build and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    dx, dockerx Docker buildx amd64 image and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    a, apply അ੤ cluster ী ߓನ೤פ׮.
    
    
    (kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. default : k8s/gcp/)
    
    
    delete, remove അ੤ cluster ী ߓನػ ղਊਸ ࢏ઁ ೤פ׮. (kubectl delete -k )
    
    
    -p, --path kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. (default : k8s/gcp/)
    
    
    l, log, logs pod ۽Ӓܳ ୶੸೤פ׮.
    
    
    e, exec kubectl exec -it {pod}
    
    
    cicd, jenkins cicd ܳ ਤೠ ౵ੌ ࣇ౴
    
    
    - Jenkinsfile ਸ ׮਍߉ই അ੤ ٣۩షܻী ݏ୾ ࣇ౴೤פ׮.
    
    
    - Dockerfile ੉ হਵݶ మ೒݁ਸ ׮਍߉णפ׮.
    
    
    - kustomization directory మ೒݁ਸ ׮਍߉णפ׮.
    
    
    ------------ Android ------------------------------------------------------
    
    
    rb-setenv remotebuildী ೙ਃೠ ssh keyܳ ࢤࢿೞҊ, podী public keyܳ ١۾೤פ׮.
    
    
    rb-portforward remotebuild podী ssh োѾਸ ਤೠ ఠօ݂ਸ ݛযસפ׮.
    
    
    rb-createpod नӏ Userо ࢎਊೡ remotebuild ਊ podܳ ࢤࢿ೤פ׮.
    
    
    ================================================================================
    
    
    KAKAO MOBILITY DEVOPS
    

    MOBIOPS
    

    View Slide

68. ================================================================================
    
    
    # # #### ##### # #### ##### ####
    
    
    ## ## # # # # # # # # # #
    
    
    # ## # # # ##### # # # # # ####
    
    
    # # # # # # # # # ##### #
    
    
    # # # # # # # # # # # #
    
    
    # # #### ##### # #### # ####
    
    
    --------------------------------------------------------------------------------
    
    
    Usage:
    
    
    mobiops [command] [options]
    
    
    Available Commands:
    
    
    --------------- GCP ------------------------------------------------------
    
    
    login ࢶఖೡࣻ ੓ח ҅੿ ܻझ౟ܳ ઑഥ೤פ׮.
    
    
    -n, --new ࢜۽਍ ҅੿ਵ۽ ۽Ӓੋ ೤פ׮.
    
    
    -g, --grep Ѩ࢝ೡ gcp ೐۽ં౟੉ܴਸ ੑ۱೤פ׮. grep -E option
    
    
    c, configure GKE ী ੽Ӕ೤פ׮.
    
    
    -r, --refresh GKE ੽Ӕ ݾ۾ਸ јन೤פ׮.
    
    
    p, project ࢜۽਍ ೐۽ં౟ܳ оܻఈפ׮.
    
    
    k, kube അ੤ ۽Ӓੋೠ ҅੿ਵ۽ ੽Ӕೡ ࣻ ੓ח GKE credential ਸ оઉ২פ׮.
    
    
    cp-secret secret ਸ copy ೤פ׮.
    
    
    usage : /usr/local/bin/mobiops cp-secret SECRET_NAME OLD_Namespace NEW_Namespace
    
    
    w, workload, workid workload identity ܳ োѾ೤פ׮. ksa, gsa োѾ
    
    
    s, ssh GCE ੋझఢझী IAPܳ ా೧ SSH ੽ࣘ೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    r, rdp GCE ੋझఢझী RDP ੽ࣘਸ ਤ೧ IAPܳ ాೠ ఠօਸ प೯೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    --------------- K8S ------------------------------------------------------
    
    
    ns, namespace namespace ܳ ࣇ౴೤פ׮.
    
    
    x, context context ܳ ࣇ౴೤פ׮. (kubectx प೯)
    
    
    b, build ࣗझ௏٘ܳ ࠽٘ ೤פ׮. (default: ./gradlew build -x test)
    
    
    -s, --spring spring build (./gradlew build -x test)
    
    
    d, docker Docker build and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    dx, dockerx Docker buildx amd64 image and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    a, apply അ੤ cluster ী ߓನ೤פ׮.
    
    
    (kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. default : k8s/gcp/)
    
    
    delete, remove അ੤ cluster ী ߓನػ ղਊਸ ࢏ઁ ೤פ׮. (kubectl delete -k )
    
    
    -p, --path kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. (default : k8s/gcp/)
    
    
    l, log, logs pod ۽Ӓܳ ୶੸೤פ׮.
    
    
    e, exec kubectl exec -it {pod}
    
    
    cicd, jenkins cicd ܳ ਤೠ ౵ੌ ࣇ౴
    
    
    - Jenkinsfile ਸ ׮਍߉ই അ੤ ٣۩షܻী ݏ୾ ࣇ౴೤פ׮.
    
    
    - Dockerfile ੉ হਵݶ మ೒݁ਸ ׮਍߉णפ׮.
    
    
    - kustomization directory మ೒݁ਸ ׮਍߉णפ׮.
    
    
    ------------ Android ------------------------------------------------------
    
    
    rb-setenv remotebuildী ೙ਃೠ ssh keyܳ ࢤࢿೞҊ, podী public keyܳ ١۾೤פ׮.
    
    
    rb-portforward remotebuild podী ssh োѾਸ ਤೠ ఠօ݂ਸ ݛযસפ׮.
    
    
    rb-createpod नӏ Userо ࢎਊೡ remotebuild ਊ podܳ ࢤࢿ೤פ׮.
    
    
    ================================================================================
    
    
    KAKAO MOBILITY DEVOPS
    

    MOBIOPS
    

    View Slide

69. ================================================================================
    
    
    # # #### ##### # #### ##### ####
    
    
    ## ## # # # # # # # # # #
    
    
    # ## # # # ##### # # # # # ####
    
    
    # # # # # # # # # ##### #
    
    
    # # # # # # # # # # # #
    
    
    # # #### ##### # #### # ####
    
    
    --------------------------------------------------------------------------------
    
    
    Usage:
    
    
    mobiops [command] [options]
    
    
    Available Commands:
    
    
    --------------- GCP ------------------------------------------------------
    
    
    login ࢶఖೡࣻ ੓ח ҅੿ ܻझ౟ܳ ઑഥ೤פ׮.
    
    
    -n, --new ࢜۽਍ ҅੿ਵ۽ ۽Ӓੋ ೤פ׮.
    
    
    -g, --grep Ѩ࢝ೡ gcp ೐۽ં౟੉ܴਸ ੑ۱೤פ׮. grep -E option
    
    
    c, configure GKE ী ੽Ӕ೤פ׮.
    
    
    -r, --refresh GKE ੽Ӕ ݾ۾ਸ јन೤פ׮.
    
    
    p, project ࢜۽਍ ೐۽ં౟ܳ оܻఈפ׮.
    
    
    k, kube അ੤ ۽Ӓੋೠ ҅੿ਵ۽ ੽Ӕೡ ࣻ ੓ח GKE credential ਸ оઉ২פ׮.
    
    
    cp-secret secret ਸ copy ೤פ׮.
    
    
    usage : /usr/local/bin/mobiops cp-secret SECRET_NAME OLD_Namespace NEW_Namespace
    
    
    w, workload, workid workload identity ܳ োѾ೤פ׮. ksa, gsa োѾ
    
    
    s, ssh GCE ੋझఢझী IAPܳ ా೧ SSH ੽ࣘ೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    r, rdp GCE ੋझఢझী RDP ੽ࣘਸ ਤ೧ IAPܳ ాೠ ఠօਸ प೯೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    --------------- K8S ------------------------------------------------------
    
    
    ns, namespace namespace ܳ ࣇ౴೤פ׮.
    
    
    x, context context ܳ ࣇ౴೤פ׮. (kubectx प೯)
    
    
    b, build ࣗझ௏٘ܳ ࠽٘ ೤פ׮. (default: ./gradlew build -x test)
    
    
    -s, --spring spring build (./gradlew build -x test)
    
    
    d, docker Docker build and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    dx, dockerx Docker buildx amd64 image and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    a, apply അ੤ cluster ী ߓನ೤פ׮.
    
    
    (kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. default : k8s/gcp/)
    
    
    delete, remove അ੤ cluster ী ߓನػ ղਊਸ ࢏ઁ ೤פ׮. (kubectl delete -k )
    
    
    -p, --path kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. (default : k8s/gcp/)
    
    
    l, log, logs pod ۽Ӓܳ ୶੸೤פ׮.
    
    
    e, exec kubectl exec -it {pod}
    
    
    cicd, jenkins cicd ܳ ਤೠ ౵ੌ ࣇ౴
    
    
    - Jenkinsfile ਸ ׮਍߉ই അ੤ ٣۩షܻী ݏ୾ ࣇ౴೤פ׮.
    
    
    - Dockerfile ੉ হਵݶ మ೒݁ਸ ׮਍߉णפ׮.
    
    
    - kustomization directory మ೒݁ਸ ׮਍߉णפ׮.
    
    
    ------------ Android ------------------------------------------------------
    
    
    rb-setenv remotebuildী ೙ਃೠ ssh keyܳ ࢤࢿೞҊ, podী public keyܳ ١۾೤פ׮.
    
    
    rb-portforward remotebuild podী ssh োѾਸ ਤೠ ఠօ݂ਸ ݛযસפ׮.
    
    
    rb-createpod नӏ Userо ࢎਊೡ remotebuild ਊ podܳ ࢤࢿ೤פ׮.
    
    
    ================================================================================
    
    
    KAKAO MOBILITY DEVOPS
    

    MOBIOPS
    

    View Slide

70. ================================================================================
    
    
    # # #### ##### # #### ##### ####
    
    
    ## ## # # # # # # # # # #
    
    
    # ## # # # ##### # # # # # ####
    
    
    # # # # # # # # # ##### #
    
    
    # # # # # # # # # # # #
    
    
    # # #### ##### # #### # ####
    
    
    --------------------------------------------------------------------------------
    
    
    Usage:
    
    
    mobiops [command] [options]
    
    
    Available Commands:
    
    
    --------------- GCP ------------------------------------------------------
    
    
    login ࢶఖೡࣻ ੓ח ҅੿ ܻझ౟ܳ ઑഥ೤פ׮.
    
    
    -n, --new ࢜۽਍ ҅੿ਵ۽ ۽Ӓੋ ೤פ׮.
    
    
    -g, --grep Ѩ࢝ೡ gcp ೐۽ં౟੉ܴਸ ੑ۱೤פ׮. grep -E option
    
    
    c, configure GKE ী ੽Ӕ೤פ׮.
    
    
    -r, --refresh GKE ੽Ӕ ݾ۾ਸ јन೤פ׮.
    
    
    p, project ࢜۽਍ ೐۽ં౟ܳ оܻఈפ׮.
    
    
    k, kube അ੤ ۽Ӓੋೠ ҅੿ਵ۽ ੽Ӕೡ ࣻ ੓ח GKE credential ਸ оઉ২פ׮.
    
    
    cp-secret secret ਸ copy ೤פ׮.
    
    
    usage : /usr/local/bin/mobiops cp-secret SECRET_NAME OLD_Namespace NEW_Namespace
    
    
    w, workload, workid workload identity ܳ োѾ೤פ׮. ksa, gsa োѾ
    
    
    s, ssh GCE ੋझఢझী IAPܳ ా೧ SSH ੽ࣘ೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    r, rdp GCE ੋझఢझী RDP ੽ࣘਸ ਤ೧ IAPܳ ాೠ ఠօਸ प೯೤פ׮.
    
    
    -r, --refresh GCE ੋझఢझ ݾ۾ਸ јन೤פ׮.
    
    
    --------------- K8S ------------------------------------------------------
    
    
    ns, namespace namespace ܳ ࣇ౴೤פ׮.
    
    
    x, context context ܳ ࣇ౴೤פ׮. (kubectx प೯)
    
    
    b, build ࣗझ௏٘ܳ ࠽٘ ೤פ׮. (default: ./gradlew build -x test)
    
    
    -s, --spring spring build (./gradlew build -x test)
    
    
    d, docker Docker build and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    dx, dockerx Docker buildx amd64 image and push to GCR (default: ./Dockerfile)
    
    
    -f, --file Dockerfile path (/Users/whoami/workdir/Dockerfile-custom)
    
    
    -t, --tag Docker image tag
    
    
    a, apply അ੤ cluster ী ߓನ೤פ׮.
    
    
    (kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. default : k8s/gcp/)
    
    
    delete, remove അ੤ cluster ী ߓನػ ղਊਸ ࢏ઁ ೤פ׮. (kubectl delete -k )
    
    
    -p, --path kustomization.yaml file ੉ ੓ח ҃۽ܳ ૑੿೤פ׮. (default : k8s/gcp/)
    
    
    l, log, logs pod ۽Ӓܳ ୶੸೤פ׮.
    
    
    e, exec kubectl exec -it {pod}
    
    
    cicd, jenkins cicd ܳ ਤೠ ౵ੌ ࣇ౴
    
    
    - Jenkinsfile ਸ ׮਍߉ই അ੤ ٣۩షܻী ݏ୾ ࣇ౴೤פ׮.
    
    
    - Dockerfile ੉ হਵݶ మ೒݁ਸ ׮਍߉णפ׮.
    
    
    - kustomization directory మ೒݁ਸ ׮਍߉णפ׮.
    
    
    ------------ Android ------------------------------------------------------
    
    
    rb-setenv remotebuildী ೙ਃೠ ssh keyܳ ࢤࢿೞҊ, podী public keyܳ ١۾೤פ׮.
    
    
    rb-portforward remotebuild podী ssh োѾਸ ਤೠ ఠօ݂ਸ ݛযસפ׮.
    
    
    rb-createpod नӏ Userо ࢎਊೡ remotebuild ਊ podܳ ࢤࢿ೤פ׮.
    
    
    ================================================================================
    
    
    KAKAO MOBILITY DEVOPS
    

    MOBIOPS
    

    View Slide

71. mobiops rb
    -
    createpod
    

    View Slide

72. mobiops rb
    -
    createpod
    

    View Slide

73. mobiops rb
    -
    createpod
    

    View Slide

74. mobiops rb
    -
    setenv
    

    View Slide

75. mobiops rb
    -
    setenv
    

    View Slide

76. mobiops rb
    -
    portforward
    

    View Slide

77. mobiops rb
    -
    portforward
    

    View Slide

78. mobiops rb
    -
    portforward
    

    View Slide

79. MOBIOPS
    Remote Build
    

    View Slide

80. MOBIOPS
    Remote Build
    

    View Slide

81. MOBIOPS
    Remote Build
    

    View Slide

82. task 유형 소요시간
    upload
    최소 18ms
    평균 0.9s
    최대 29s
    execute
    최소 0.1s
    평균 45.9s
    최대 3m 25s
    download
    최소 0.1s
    평균 10.5s
    최대 1m 37s
    

    View Slide

83. task 유형 소요시간
    upload
    최소 18ms
    평균 0.9s
    최대 29s
    execute
    최소 0.1s
    평균 45.9s
    최대 3m 25s
    download
    최소 0.1s
    평균 10.5s
    최대 1m 37s
    

    View Slide

84. 환경 빌드시간 로컬빌드 리모트빌드
    증분 빌드
    최소 3 s 8s
    평균 60s 29s (-51%)
    최대 3m 47s 1m 17s
    풀 빌드
    최소 14m 30s 3m 29s
    평균 15m 23s 3m 47s (-75%)
    최대 16m 7s 4m 1s
    3 99 195 388285581 870967
    8 31 54 93 132 155 178 209233
    로컬 빌드
    리모트 빌드
    

    View Slide

85. 환경 빌드시간 로컬빌드 리모트빌드
    증분 빌드
    최소 3 s 8s
    평균 60s 29s (-51%)
    최대 3m 47s 1m 17s
    풀 빌드
    최소 14m 30s 3m 29s
    평균 15m 23s 3m 47s (-75%)
    최대 16m 7s 4m 1s
    3 99 195 388285581 870967
    8 31 54 93 132 155 178 209233
    로컬 빌드
    리모트 빌드
    

    View Slide

86. RAM 30Gb
    Throttling
    

    View Slide

87. nexus path bug test suite bug compose preview
    
    
    bug
    

    View Slide

88. nexus path bug test suite bug compose preview
    
    
    bug
    

    View Slide

89. nexus path bug test suite bug compose preview
    
    
    bug
    

    View Slide

90. Jenkins CI
    branch stability
    Remote Build
    ~69% build performance
    

    View Slide

91. Jenkins CI
    branch stability
    Remote Build
    ~69% build performance
    

    View Slide

92. branch 안정성을 검증 해주는 Jenkins CI 구축과정
    
    
    빠른 빌드를 도와주는 리모트 빌드
    
    
    향후 계획
    

    View Slide

93. View Slide

94. View Slide

95. 참고 문헌
    1) https:/
    /github.com/Adambl4/mirakle
    
    
    2) https:/
    /github.com/sonatype
    
    
    3) https:/
    /www.jenkins.io/
    
    
    4) https:/
    /www.jenkins.io/artwork/
    
    
    5) https:/
    /gradle.org/
    
    
    6) https:/
    /gradle.com/brand/
    
    
    7) https:/
    /github.com/kubernetes
    
    
    8) https:/
    /www.docker.com/legal/trademark
    -
    guidelines/
    
    
    9) https:/
    /slack.com/intl/ko
    -
    kr/media
    -
    kit
    
    
    10) https:/
    /github.com/logos
    

    View Slide

96. E.O.D
    

    View Slide

97. E.O.D
    

    View Slide